United States      Office of Policy,
Environmental Protection  Economics and Innovation
Agency        (1807T)
                      November 2009
                      EPA-100-R-10-001
Evaluation of the
Effectiveness of an
Ethanol Compliance
Manual
Final Report
 Promoting Environmental Results
 •4	
 Through Evaluation

-------
                 Acknowledgements
This evaluation was performed by Ross & Associates Environmental Consulting, Ltd. (Ross
& Associates) under contract to Industrial Economics, Incorporated (lEc) for EPA's Office of
Policy, Economics and Innovation (OPEI) under Contract EP-W-07-028 between EPA and
I EC. The evaluation team included Anna Williams, Tamara Power-Drutis and Amy Wheeless
of Ross & Associates, Ashley Betts and Ward Burns of EPA Region 7 and Michelle
Mandolia of OPEI's Evaluation Support Division.

This report was developed under the Program Evaluation Competition, sponsored by OPEI.
To access copies of this or other EPA program evaluations, please go to EPA's Evaluation
Support Division's website at http:www.epa.gov/evaluate.

-------
Table of Contents
Executive Summary	i

I.   Introduction	1
     Background	1
     Purpose of the Evaluation	2
     Report Organization	2
II.   Methodology	3
     Evaluation Approach	3
     Information Collection and Analysis	4
III.  Findings	8
     Topic Area I: Outreach and Access	8
     Topic Area II: Effectiveness of the Manual as an Information Tool	11
     Topic Area III: Changes in Understanding and Behavior	14
     Topic Area IV: Improving the Manual	16
     Topic Area V: Other Compliance Assistance	19
IV.  Recommendations	20
Appendices (Available in Separate Document)
   A.  Customer Satisfaction Survey
   B.  Customer Satisfaction Survey Results
   C.  Interview Guide - Ethanol Facility
   D.  Interview Guide-Contractor
   E.  Interview Guide - State Compliance Staff
   F.  Interview Guide - EPA Staff

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                       Final Report
Executive Summary

Background and Evaluation Process
Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were
issued new environmental permits. The increase in new ethanol facilities was especially significant given
that, during the previous 20 years, only 30 ethanol plants had been permitted.  Region 7 recognized that
the ethanol  production sector  may  need targeted  outreach and education  on the pertinent  federal
environmental rules, regulations, and programs that govern the sector's construction and operation.
To meet this need,  in  November 2007, Region 7 staff published a compliance assistance manual for new
ethanol  facilities entitled Environmental Laws Applicable to Construction and Operation of Ethanol Plants
(hereto  after referred  to as "the Manual").  The Manual included information  on federal environmental
programs and the  roles of state and federal authorities in overseeing ethanol facility construction and
operation.  The  Manual was  primarily targeted to ethanol plants, but was also likely of interest to state
regulatory  program offices,  trade associations, consultants, and the public.   Region 7  conducted a
number of outreach activities to  publicize the Manual to ethanol plants and other stakeholders.
Region  7,  with support from the EPA Office of  Policy, Economics and  Innovation (OPEI), wished to
evaluate the success  of the Manual in improving industry compliance with relevant rules and regulations
and to gather information on the Manual's readability, quality of information, and overall usefulness as a
compliance assistance tool to ethanol facilities in the Region. In addition, Region 7 wanted to  identify
ways to improve facility satisfaction with the Manual as a compliance assistance tool and  to determine
what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of
this evaluation are  intended  primarily for Region 7 staff as they prepare  revisions to the  Manual and
produce other outreach  materials for the ethanol production sector.
An EPA-contractor  team led the evaluation.  Two representatives from Region 7 were part of the team;
one who had helped to develop the  Manual as  a member of the Region 7  Biofuels Team, and one
representative of the  Region's  policy and management division.  The other EPA team member was an
evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI. The contractor team
                      Exhibit ES-i. Overarching Evaluation Questions
   Topic Area I: Outreach and Access.  How effective are Region 7's efforts to advertise the
   availability of and distribute the Manual to ethanol facilities and other intended audiences?

   Topic Area II:  Effectiveness of the  Manual as an Information  Tool.  How effective is the
   Manual as a tool for transferring compliance (and related facility technical support) information to
   ethanol facilities?

   Topic Area III:  Changes in Understanding and Behavior. Has the existence and use of the
   Manual led to increased understanding and behavior change to support improved compliance?

   Topic Area IV:  Improving the Manual. How could the Manual be improved to provide effective
   compliance assistance to ethanol facilities?

   Topic Area V: Other Compliance Assistance.  What approaches other than the  Manual would
   provide effective compliance assistance to the ethanol  facilities?
November 2009                                                                            ES-i

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
consisted  of Industrial  Economics  (lEc)  and  a subcontractor, Ross & Associates Environmental
Consulting,  Ltd. (Ross  &  Associates).    Ross  & Associates  implemented most  of the contractor
responsibilities and produced this report with support from the other team members.
The evaluation focused on five overarching evaluation questions, listed on  the previous page in Exhibit
ES-1.   To answer the  overarching  questions, the  evaluators  conducted 20  interviews  with facility
environmental managers, ethanol industry  contractors, state compliance staff members, and EPA staff
members.  In addition, the  evaluators developed a customer satisfaction survey and distributed the
survey to  ethanol facilities in Region  7 and ethanol industry contractors known  to work in  the Region.
Finally, the evaluators conducted research on ethanol plant compliance to try to answer the overarching
evaluation question on the effect the Manual had on facility compliance.
Findings
Those interviewed and surveyed for the evaluation found the Manual to be very useful.  They appreciated
its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the proactive
assistance  approach embodied  in the  Manual  represented.  Generally,  comments for improvement
focused on timelier and improved distribution and a variety of changes to make the Manual a stronger
compliance assistance tool.  Below were the major findings for each overarching evaluation topic area.

Topic Area I: Outreach and Access
Awareness of the Manual amongst the primary audiences—ethanol facilities and contractors—was not as
prevalent as it could be despite outreach efforts by Region 7 staff.  In addition, the timing of the Manual's
publication occurred after most new plants had commenced or completed construction; only  a handful of
new ethanol  plants started productions since November 2007.  A majority of the  facilities that the
evaluators spoke with in the summer of 2009 were not immediately aware of the  Manual's existence.
Nonetheless, most of these facilities were receptive to reviewing the Manual as a part of this evaluation.
Feedback also indicated that the audience for the Manual is larger  than just Region 7-specific ethanol
facilities and contractors, and that promoting the Manual nationally could be beneficial, particularly to EPA
Regions with significant numbers of ethanol plants.

Topic Area II: Effectiveness of the Manual as an Information Tool
Participants in the evaluation found the Manual to be well organized, easy to navigate, and  an appropriate
tool  for conveying compliance information to facilities.   Having this  material, including  appropriate
contacts, organized in  one place provided for an  easy reference for facility managers and  they found the
Manual to be readable and  organized logically.  Participants provided varied and inconsistent feedback
on whether the Manual was too technical or not technical enough, but everyone interviewed and surveyed
indicated that they found the Manual to be useful in some way. Whether the Manual has the appropriate
level of detail likely depends on the reader's existing level of knowledge and facility needs.

Topic Area III: Changes in Understanding and Behavior
All of the facilities and contractors that participated in this evaluation said the  Manual was comprehensive
and  was helpful  for  increasing  or  solidifying their understanding  of  environmental  compliance
requirements.
This evaluation cannot conclusively say that  the Manual has led to  any significant changes that would
affect facility  compliance. With only a few exceptions, respondents to this  evaluation were not able to
identify specific instances where  facilities had more awareness of applicable federal regulations or had
made compliance or management changes as a result of the Manual.  Research of the EPA Enforcement

November 2009                                                                            ES-ii

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report


and Compliance History Online (ECHO) database showed that, after the Manual was published in late
2007, there was some change in informal and formal enforcement actions at Region 7 ethanol facilities,
but due to  lagging and incomplete enforcement data and the inability to know  exactly when a facility
reviewed the  Manual,  the  evaluators cannot  say conclusively than the Manual had an effect  on
compliance.

Topic Area IV: Improving the Manual
Respondents to this evaluation had a number of suggestions on how future versions of the Manual could
be improved.  These suggestions  centered  around additional content, such as  a section  on cellulosic
ethanol, or around functional content, such as checklists, case  studies, and  specific  examples.  The
recommendations below  carry forward many of the suggestions offered  by those  interviewed  and
surveyed.

Topic Area V: Other Compliance Assistance
Evaluation  participants  thought that, while the Manual itself was a useful tool,  other venues, such as
conferences, EPA workshops,  a more interactive  Web site, and one-on-one interaction  with ethanol
facilities were also valuable approaches for providing compliance assistance to ethanol facilities.

Recommendations

Evaluation participants clearly conveyed that the Manual was useful and valuable and that it effectively
communicated relevant information for ethanol plants.  The following recommendations focus on ways to
improve the Manual for any potential  future  versions and on other ways that Region 7 could make the
existing information  available to its intended audiences.  The evaluators understand that there may be
limited resources available for updating the  Manual and limited time available for staff  to provide other
kinds of compliance assistance to ethanol facilities.  Further, the market for ethanol has been shifting
dramatically and demand for new facility permits has not returned to the peak it was in when the Manual
was  first released.  Demand for  the Manual  and feasibility of  implementing any  of  the evaluation's
recommendations will likely  be  influenced by these and other factors. The following recommendations,
which are expanded upon in the body of the report, are offered for Region 7's consideration.
1.  As a first step, consider whether to tailor the Manual's content and distribution to the national level.
2.   Consider additional  outreach strategies and targeted audiences.
3.   Consider making this tool primarily Web-based, with a hard copy Manual as a supplement.
4.   Update the Manual to include new regulatory and compliance information.
5.   Create an executive summary to encapsulate the main points of the Manual.
6.   Incorporate more "at a glance"  information, such as lists of resources, call-out boxes, checklists, and
    calendars.
7.   Provide more specific examples and case studies.
8.   Expand content to include regulatory information about topics of interest to ethanol plants, such as
    cellulosic ethanol production and byproduct diversion into other products.
9.   Expand information on beyond  compliance efforts and best practices.
10. Apply the  lessons from this evaluation  to the Region's other  compliance assistance  and beyond
    compliance efforts.


November 2009                                                                             ES-iii

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
I.  Introduction

Background

Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were
issued new environmental permits. The increase in new ethanol facilities was especially significant given
that, during the previous 20 years, only 30 ethanol plants had been permitted.  Region 7 recognized that
the ethanol production  sector may need  targeted outreach and education to understand the  pertinent
federal  environmental  rules,  regulations, and  programs  that govern  the  sector's  construction and
operation.
To meet this need, in November 2007,  Region 7 staff published a
compliance  assistance  manual  for new ethanol facilities  entitled    "The Manual in  general was
Environmental Laws Applicable  to  Construction and  Operation  of    a huge  undertaking  and  a
Ethanol Plants (hereto after referred to as "the Manual").  The Manual    great  idea... a  good  public
included information on federal environmental programs and the roles  |  service..." - EPA contact
of state  and  federal  authorities  in  overseeing  ethanol  facility
construction and operation.1
Ethanol plants were the  primary audience for the Manual,  but the  Manual was also likely of interest to
state regulatory program  offices, trade associations, consultants and the public.  To distribute the Manual
to the relevant stakeholders, Region 7 staff conducted a number of outreach activities:
•  Staff announced the  development of the  Manual at the 2007  National  Association  of  Farm
   Broadcasting Convention and  distributed ethanol materials to attendees and exhibitors.
•  Staff developed a news release about the development of the Manual and distributed it to those who
   had expressed  interest and to  the editors of agriculture and biofuels magazines.
•  Staff posted the Manual on the EPA Region 7 Web site.
•  The Region 7 Office  of Public Affairs (OPA) shared ethanol postcards with the Chairs of the Missouri
   House Agriculture Committee, the Agribusiness Committees, and the Senate Agriculture Committee,
   along with members  of these committees, as well as at relevant conferences (e.g., Commodity Class,
   the  Missouri Agribusiness Association  annual  meeting, the Kansas  Feed  and  Grain annual
   conference, the Nebraska Governor's Conference  on Agriculture,  and the Missouri Governor's
   Conference on  Agriculture).
•  The Region  7  Regional Administrator (RA) and OPA attended the Fuel Ethanol  Workshop and
   announced the  development of the Manual.  OPA collected contact information for those interested in
   the Manual and notified them via email when Region 7 released  the Manual.
•  The RA conducted  news  interviews  about the  Manual (e.g.,  Missourinet, WHO-Farm radio) and
   mentioned the  Manual in a number of speeches (e.g., at Commodity Classic,  Bunge North America,
   Inc., CenSARA's National Environmental Biofuels Conference).
•  OPA met with executives of ethanol plants and shared information about the Manual.
1 A copy of the Manual can be downloaded at:
http://www.epa.qov/reqion07/priorities/aqriculture/ethanol plants manual.pdf

November 2009                                                                           Page 1

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
    Air program staff discussed the Manual at a number of events and meetings (e.g., Iowa State Field
    Briefing, a meeting with the head  of EPA's  Office of Enforcement and Compliance Assistance,
    CenSARA Director's Meeting).
    Staff informed a  number of interested stakeholders about the Manual, including RCRA  (Resource
    Conservation and Recovery Act)  State programs, the U.S.  Army  Corps of Engineers.  Staff also
    informed two ethanol facilities about the Manual during the facilities' RCRA inspections.
    Each Emergency Planning  & Community  Right-to-Know Act (EPCRA) state coordinator announced
    the availability of the Manual at his or her State Emergency Response Commission meetings.
Purpose of the Evaluation
Region 7, with support from the EPA Office of Policy,  Economics, and Innovation (OPEI), wished to
evaluate the success of the Manual in improving industry compliance with relevant rules and regulations
and to gather information  on the Manual's readability, quality of information, and overall usefulness as a
compliance assistance tool  to ethanol facilities in the Region.  In addition,  Region 7 wanted to  identify
ways  to improve facility satisfaction with the Manual as a compliance assistance tool and to determine
what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of
this evaluation are intended primarily for  Region 7 staff as they prepare revisions to the Manual  and
produce other outreach materials for the ethanol production sector.
An EPA-contractor team led the evaluation.  Two representatives from Region 7 were  part of the team;
one who had helped to develop the Manual as  a member of the Region 7  Biofuels  Team, and  one
representative of the Region's policy and management division. The other EPA team member was an
evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI.  The contractor team
consisted  of Industrial  Economics  (lEc)  and  a subcontractor,  Ross  &  Associates Environmental
Consulting,  Ltd.  (Ross  &  Associates).   Ross  & Associates  implemented  most of the  contractor
responsibilities and produced this report with support from the other team members.
The results of this  evaluation are intended primarily for Region 7 staff as they prepare revisions to the
Manual  and produce other outreach material for the ethanol production sector.   The results  of this
evaluation could also assist this and other Regions with compliance assistance materials for industry and
manufacturing sectors.
Report Organization
This report consists of four sections, in addition to this Introduction (Section I).  Section II provides an
overview of the evaluation  methodology.  Section III describes the findings for each evaluation question.
Section IV lists and describes the evaluation's concluding recommendations for Region 7.  Appendices,
which  include  a copy of the  survey,  the survey  results, and the interview guides,  are provided in a
separate document.
November 2009                                                                           Page 2

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
II.  Methodology
The following section provides  an overview of the  evaluation methodology, including the team's initial
approach for planning the evaluation and information collection and analysis process.

Evaluation Approach

As a first step in the evaluation,  the evaluation team refined the initial evaluation questions that had been
submitted as  a part of OPEI's  evaluation competition,  identified the information needs to answer the
refined questions, and updated the draft ethanol manual  logic model that had been developed previously
to help inform the evaluation.
Evaluation Questions
The  evaluation team developed  the  overarching evaluation questions listed below to structure  the
evaluation. The  overarching  questions cover five topics: effectiveness of distribution of to the Manual,
effectiveness  of the Manual as  an information tool, effectiveness of the Manual on understanding  and
behavior changes, possible improvements to the Manual, and other approaches for providing compliance
tools.
                        Exhibit i. Overarching Evaluation Questions
     Topic Area I: Outreach and Access.  How effective are Region 7's efforts to advertise the
     availability of and distribute the Manual to ethanol facilities and other intended audiences?
     Topic Area II: Effectiveness of the Manual as an Information Tool.  How effective is the
     Manual as a tool for transferring compliance (and related facility technical support) information
     to ethanol facilities?
     Topic Area III: Changes  in Understanding and Behavior. Has the existence and use  of
     the  Manual  led  to  increased  understanding and  behavior change to support improved
     compliance?
     Topic Area  IV: Improving  the Manual.  How could the Manual be improved to provide
     effective compliance assistance to ethanol facilities?
     Topic Area V: Other Compliance Assistance.  What approaches other  than the Manual
     would provide effective compliance assistance to the ethanol facilities?
These overarching evaluation questions informed data collection, including development of the interview
guides and the survey (see Appendices).  This  report presents findings according to each overarching
evaluation question.

Evaluability Assessment
Following  development of the evaluation questions, the evaluation  team conducted an  evaluability
assessment to ensure that there would be enough information to assess the effectiveness of the Manual.
This evaluability assessment identified:
•   What information would be needed to address the key evaluation questions;
•   Whether the needed information would be available; and
•   How the available information would be collected.

November 2009                                                                          Page 3

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
The conclusion of this evaluability assessment was that, with the available data and information collection
methods, to some extent all evaluation questions would be answerable, with qualitative, yet authoritative,
information and stakeholder feedback.   The  one  exception  was  Topic Area  III:  the  evaluability
assessment concluded that, due to a  lack of data, the evaluation would not be  able to draw definitive
conclusions about a causal relationship between the Manual and facility compliance.

Logic Model
In preparing for this evaluation, Region 7 developed a logic model for the Manual and  its associated
activities.  The evaluation team refined the model to elaborate on potential outcomes of Manual use (see
Exhibit 2 on the following page). The final logic model depicts the linkages between program activities
inputs, outputs, stakeholders, and expected outcomes of Region 7's outreach activities and compliance
assistance tool.
Information Collection and Analysis
In planning for the evaluation, the evaluation team determined that interviews with relevant stakeholders,
supplemented by a survey of facilities and contractors, would help to answer the overarching evaluation
questions.  The primary target audience for the Manual and thus  the main source of feedback were
ethanol facilities and contractors.  The secondary audiences were state and EPA  Region 7 compliance
staff.  While trade associations and the  public were also listed in the logic  model as an "audience" or
"customers" for the Manual, they were not targeted for separate information collection in this evaluation.
Using information provided by Region 7, the evaluators contacted the ethanol  plants and a handful of
contractors to assess interest in participating in an interview and/or a survey.  During this contact, the
evaluators also offered to send Internet links for the Manual to those facilities and  contractors who were
not previously aware of the Manual or had not received it.
In addition to  interviews  and surveys,  the  evaluators  conducted some  limited  research  on  facility
compliance in the Region to better understand any changes in facility compliance.

Interview Approach
The evaluators collected  data from ethanol plants, contractors, EPA staff,  and state compliance staff
through a set of telephone  interviews.  Facility and  contractor interviews  were  with individuals who
confirmed that they  had received and reviewed the Manual.  Region 7 staff contributed suggestions on
who to interview at EPA and state agencies. The list of interviewees was reviewed  by other members of
the evaluation team to ensure a representative selection. The final set of interviewees consisted of:
•   Individuals at seven facilities that had responsibilities pertaining to environment compliance.
•   Three ethanol-industry environmental  consultants/contractors;  an  additional  consultant submitted
    written feedback.
•   Five state compliance staff members  from three Region 7 state environmental agencies.
•   Five EPA staff members, four of whom were from Region 7 and one from EPA Headquarters.
November 2009                                                                            Page 4

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
                                        Exhibit 2. Ethanol Compliance Manual Logic Model
Goal: Improve environmental regulatory compliance among ethanol facilities in order to reduce impacts on environmental and human health



Resources from:
• EPA Region 7
Biofuels Team
• EPA Region 7
Ag Team
i State
Agencies
• Trade Groups












ix
ACTIVITIES
• Create
Compliance
Manual
• Answer
Regulatory
Questions
• Develop Other
Outreach
Materials
• Improve and
Revise Manual








PARTICIPANTS
• Ethanol
Facilities
• Biofuels Team
and Regulatory
Programs
• State
Compliance
Staff
• Ethanol
Contractors
• Trade Groups

i






ix
SHORT TERM
Improved Awareness and
Knowledge of Regulatory
Requirements and
Attitude Change


OUTCOMES
MEDIUM TERM
Changes in Ethanol
Facilities Operations or
Management



LONG TERM2
Improved Facility
Compliance

Reduced
Environmental and
Human Health
Impacts
OUTCOME MEASURES
# of facilities that say they
better understand how to
comply with regulations
# of facilities that
contacted someone for
further compliance
assistance
# of Manuals accessed
(print copies/ downloads)



# of facilities that took at
least one recommended
action to comply with
regulations
# of facilities that
adopted process
changes
# of facilities improving
environmental
management systems or
conducting reviews

Compliance rate
changes
#of facilities that
have changed
regulatory status
#of facilities that
eliminated, treated or
reduced emissions or
other pollutants
Quantified
environmental
improvements
 It was assumed that increased compliance with environmental laws would result in the long-term outcomes of improved environmental and public health and that
these long-term outcomes would be realized over the course of months or years.  It was beyond the scope or ability of this evaluation to identify whether these
outcomes have occurred as a direct result of the compliance assistance manual in question. Similarly,  it was not feasible as part of this evaluation to gather the
data needed to develop the long-term (aspirational) performance measures listed in the logic model.
November 2009
     Page 5

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
Interviewees received interview guides (see Appendices C, D, E, and F) in advance of their interviews.
Interviews were generally conducted by  one Ross  & Associates  staff member and lasted 15 to 30
minutes. The evaluators conducted the telephone interviews based on the interview guides; however, the
order in which  topics were discussed sometimes varied.  Notes from these interviews were collected
during the interviews and entered in an Excel workbook. These notes were considered confidential and
not included in this report.  Any specific quotes that this report  uses as examples were sent to the
relevant interviewees to ensure accurate communication of ideas, and are not attributed to the individual
in this report.
The evaluation  team had originally  considered conducting  a set of follow-up interviews with the  primary
and secondary audiences after doing an initial round of information collection.  However, after conducting
the interviews,  the  evaluation team  determined that follow-up interviews would be unlikely to yield
additional feedback or insight. However, the evaluation team did meet with members of the EPA Region
7 Biofuels Team to discuss the report's draft findings and recommendations.

Survey Approach
There were  as many as 150 ethanol facilities in Region  7,  as well as a number of ethanol industry
contractors, who may have received and read the Manual. To reach as many of them as possible and
understand their  access to and satisfaction with  the  Manual, the evaluation  team developed  and
administered a written customer satisfaction survey to be completed by mail or on the Internet.
The Federal Paperwork Reduction Act of 1995 requires  that federal agencies receive approval from the
Office of Management and Budget (OMB) before requesting information from more than nine non-federal
entities, and must do so by submitting an Information Collection Request (ICR) to OMB for approval.  The
evaluation team developed a survey, submitted the survey under generic ICR 1711.05 to OMB,  and
received approval following some minor survey revisions.
Following OMB approval of the survey, Ross & Associates emailed or mailed the survey to all Region 7
ethanol  facilities and contractors for  which  it  had contact information.   This survey (see Appendix A)
focused  on  customer satisfaction with  the  Manual.   Survey responses  were  anonymous  unless
respondents chose to identify themselves.
Participants were encouraged to respond to the survey within  approximately four weeks.  Approximately
three  weeks after the survey was mailed, the  evaluators contacted facilities and  contractors  with a
reminder email or reminder postcard and extended the due date to solicit more responses.
The evaluators mailed or emailed  surveys  to approximately  120  facilities and contractors.   Of these,
approximately ten were returned as having no recipient. Of the remaining, 24 facilities and contractors
completed surveys;  at least three  had  also participated in a telephone interview.  Others may have
participated in telephone interviews, but did not identify themselves in the survey.
The response rate for the survey was approximately  20  percent; generally, response rates for customer
satisfaction surveys range from 10 percent to 30 percent, so this  response  rate was not unexpected.
Factors  that may have contributed to this  response rate  include the fact that  the  evaluators  did not
connect with all facilities and contractors to  alert a specific contact that a survey would be forthcoming;
some facilities  in Region  7 were  no longer in  operation; and there was no tangible  and immediate
incentive offered for completing the survey.

Additional Information Review
In addition to the  surveys and interviews, the evaluators reviewed background material and researched
compliance information through available data to  help answer the overarching  evaluation questions.  The
evaluators reviewed  the  Manual  itself, the  outreach  activities that Region 7  had done to promote

November 2009                                                                            Page 6

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
distribution  of  the   Manual,  and   the   plant  list  available  at  Ethanol  Producer  Magazine
(http://ethanolproducer.com/plant-list.isp'), which provides status of operation,  when the facility started
production, and type of feedstock.  In  addition, the evaluators conducted Internet searches to see which
organizations or companies had posted links to the Manual from their Web sites.
Finally, the evaluators reviewed EPA's Enforcement and Compliance History Online (ECHO) database.
ECHO provides an overview of a facility's environmental record under the Clean Air Act (CAA) Stationary
Source Program, the Clean Water Act (CWA) National  Pollutant Elimination Discharge System (NPDES),
and the  Resource Conservation and Recovery Act (RCRA), and associated state and federal responses.
The evaluators queried each facility in  Region 7 that had information in the database and tracked informal
and formal violations pre-publication of the  Manual and post-publication of the  Manual (the Manual was
published  in November 2007).  The evaluators also looked at the compliance history of specific facilities
that were interviewed or surveyed and  compared the results for a richer analysis of compliance.
November 2009                                                                           Page 7

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
        Findings
The  following pages describe  the evaluation's key findings, organized  by the overarching evaluation
questions. Those interviewed and surveyed for the evaluation found the Manual to be very useful.  They
appreciated its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the
proactive  assistance  approach  embodied  in  the Manual  represented.    Generally, comments for
improvement focused on timelier and improved distribution and a variety of changes to make the Manual
a stronger compliance assistance tool.

Topic Area I: Outreach and Access

How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol
facilities and other intended audiences?

Overarching Findings
As discussed in Section  I, Region  7 staff conducted a  number of outreach  activities to publicize the
availability of the Manual to ethanol plants and other interested stakeholders. However, awareness of the
Manual was not as prevalent as it could be for these audiences, and the timing  of its publication was not
coincident with the  largest increase in construction of ethanol plants. Nonetheless, access to the Manual
was  sufficient for stakeholders  who were aware of it, and many had additional suggestions for how to
improve advertisement of the Manual.   Feedback also indicated that the audience for the  Manual was
larger than just  Region 7-specific  ethanol facilities  and contractors, and  that promoting  the Manual
nationally could be  beneficial, particularly to EPA regions with significant numbers of ethanol plants.
•  /Access to Manual: Overall, for facilities and contractors aware of the Manual, access to the Manual
   was considered sufficient, with many receiving it from an EPA or state contact or at a  workshop or
   conference.  However, a majority  of facilities (31 of 51) that the evaluators spoke to  in the  initial
   evaluation contact period were not  immediately aware of the Manual's existence.  Nonetheless, most
   of these facilities were receptive to receiving a Web link to the Manual.  Of these 51 facilities, eight
   had started  production in 2008 or 2009; only one of these newer facilities indicated awareness of the
   Manual prior to the evaluation.
•  Timing: Many interviewees noted  that the timing  of  publication of the  Manual was not ideal—few
   ethanol facilities have been constructed since November 2007, after the Manual was published, and
   many have gone out of business or halted production since that time.
•  Distribution Audiences: Many facilities indicated that they use contractors and consultants to find out
   information  on current or  upcoming  regulations  and refer to
    them with questions on compliance or permitting.  Facilities also  I "Ethanol is not unique to Region
    look to other ethanol-related organizations that interact regularly   7   |f a regjon comes  out with
    with the ethanol sector for information.   These groups were   something    that's   a  good
    noted as venues for distribution if EPA were to develop future   product, then the agency should
    versions of the Manual.                                        consider  marketing  it  beyond
    National  Relevance:  Several  interviewees  noted  that  the   tnat region... It  should be  a
    information  in the Manual would be well-received in other EPA   national manual.   The federal
    Regions with significant numbers of ethanol plants.  In addition,   regulations are all the same, so
    research  during  this evaluation  found  that the Manual was   tney should  all  use the same
    linked on the state environmental agency Web sites of states   manual."-EPA contact
    not in Region 7,  highlighting  that non-Region 7 ethanol plants
November 2009                                                                            Page 8

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                         Final Report


    may have found this information to be relevant.

Interview Findings
Of those that participated in the interviews, six of the seven facilities and two of the three contractors were
aware of the Manual prior to being contacted for the evaluation. These interviewees indicated that they
had received it from: a contact at EPA Region 7; a manager; training sessions and workshops; and an
environmental consultant.
Generally, the timing of the publication of the Manual was more of an issue than mode of distribution in
getting information out to relevant stakeholders. As few ethanol plants have been  placed  into operation
since the Manual was published, many of the facilities the evaluators spoke with were already aware of
the information, having been in operation for some period of time.  Region 7 noted that more than 100
facilities have  been issued environmental permits between June 2005 and June  2008.  According to
Ethanol Producer Magazine,  only 11  facilities have  been  put into operation  after November  2007,
indicating that the largest increase in activity was  prior to publication of the  Manual.3  Additionally,  a
number of plants have gone out of business or have halted production  in recent months.  According to
Ethanol Producer Magazine, only seven facilities in Region 7  were not operating during the summer and
autumn of 2009.   However,  during the evaluation,  the evaluators encountered disconnected  lines,
returned surveys, and comments from ethanol  plant staff that indicated that the number of facilities that
had at least temporarily halted production was higher.
Interviewees from the state agencies  all work in some capacity with  ethanol facilities and noted that they
have not referred any facilities to the Manual,  but indicated that they may in  the  future as the ethanol
industry rebounds and new facilities are constructed. Both EPA and state staff noted that the information
in the Manual  was helpful to operators just  beginning to plan construction of a facility, but those people
are difficult to identify unless they approach regulators before beginning plant construction.
For improving distribution of the Manual, contractors were  highlighted  as an important and oft-used
resource for ethanol facilities  on environmental issues.  Five of the  seven  ethanol plants that were
interviewed  indicated that they receive assistance on  environmental compliance and permitting issues
from contractors and that these contractors alert them of new and upcoming regulations.  All of the
contractors  also  stated that they help many ethanol  plants in  Region 7  with  their environmental
compliance issues.
In addition to contractors and consultants, facility interviewees noted that they received information  about
environmental regulations and  tools from many different sources, and suggested these venues as ways
to get information about to this  audience in the future.  Region 7 distributed the  Manual using some  of the
below approaches (e.g., making information about the Manual available at workshops, alerting relevant
stakeholders),  but feedback suggested that  wider distribution  through a number of  venues, including the
following, would be useful:
•   Make Manual available at environmental workshops.
•   Provide  Manual during EPA or state on-site technical assistance  or inspections.
•   Make information available on a more interactive EPA Web site.
•   Distribute information on EPA and external email listservs.
•   Post Manual on state environmental agency Web sites.
•   Send hard copies of Manual to all potentially relevant organizations or facilities, including:
3 Ethanol Producer Magazine, Plant List. Accessed at http://ethanolproducer.com/plant-list.isp on August 18, 2009.

November 2009                                                                             Page 9

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
         Final Report
    o  Directly to contractors and consultants.
    o  State regulatory agencies.
    o  State ethanol boards.
    o  EPA program offices.
    o  National and state ethanol trade associations.
    o  National and state environmental consulting trade associations.
    o  Air resource agencies.
Finally, two  EPA staff members and  two contractors noted,  unprompted, that the information in the
Manual  is not just  relevant to  Region  7  ethanol plants, and the  Manual  should  be distributed  and
advertised more nationally.

Survey Findings
Overall, survey respondents were satisfied with their access to the Manual and had generally heard about
it either directly from EPA, from a state  contact, or during a workshop  or conference.
Four respondents, or about 17 percent of respondents, noted that they learned of the Manual specifically
when contacted with the survey  or during the initial evaluation contact period. Of the other respondents,
they had learned about the  Manual through visiting EPA's Web site,  through their  state contact, at
conferences or workshops, or from an EPA postcard.  None had learned about  it  through  an  EPA
television or radio interview, and only one due to an EPA news release (see Figure below). All but one of
the respondents, or 95 percent, indicated that they were satisfied or very satisfied with their access to the
Manual; the other respondent was neither satisfied nor dissatisfied.
                         How Respondents Learned About the Manual
                                               N=23
                               Don't Know/
                              Don't Remember
                                   4%
              Other
          Correspondence
            f-c-r E=A
               9%
EPA news relesse
     4%
November 2009
               Page 10

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report


Other Findings
The evaluators researched where this Manual had been referenced on the Internet. Region 7 conducted
a number of outreach activities to announce the publication of the Manual, including developing a press
release.   News aggregation and industry-focused Web sites and journals picked up this release and
developed short news stories  for their readers, including Biofuels Journal4, Enviro Business and Legal
Reports5, Bulk Transporter6, and ChemAlliance7.
In addition, the Manual can be found on the Nebraska Ethanol Board's and Nebraska State Department
of Environmental Quality's Web sites8 9, and on the Web sites of some state departments that are not in
Region 7, including Kentucky10 and Minnesota.11  The Minnesota Web site noted that, while the Manual
"was not directly intended for Minnesota ethanol plants, many of the same policies and regulations apply."
The  Manual had  received 129,337  Web server  requests  ("hits")  from its release  in November 2007
through August 2009.

Topic Area II: Effectiveness of the Manual as an Information Tool

How effective  is the Manual as a tool for transferring compliance (and related facility technical support)
information to ethanol facilities?

Overarching Findings
Participants in the evaluation found the Manual to be well organized, easy to navigate, and an appropriate
tool  for  conveying  compliance  information to facilities.   Having this material,  including appropriate
contacts,  organized  in one  place  provided for an  easy reference for facility managers.  Participants
indicated that they found the Manual to be readable and organized logically. Participants  provided varied
and  inconsistent  feedback on whether  the  Manual was too technical or not  technical enough,  but
everyone interviewed and surveyed  indicated that they found the  Manual to be useful in some way.
Whether the Manual has the appropriate level of detail  likely depends on the reader's existing level of
knowledge and facility needs.
•   Readable  and Organized:  Overall,  respondents for this evaluation found the  Manual to be readable
    and well-organized.
4 "EPA Ethanol Manual: Environmental Compliance Manual for Ethanol Plants."  Biofuels Journal.  May/June 08,
page 44. Accessed on September 10, 2009: http://vwwv.nxtbook.com/nxtbooks/GJ/BJ_56_08/index.php
5 "EPA Region 7 Announces Rollout of Environmental Manual for Ethanol  Facilities." Enviro Business and Legal
Reports. November 16, 2007. Accessed on August 19, 2009: http://enviro.blr.com/news.aspx?id=83282
6 "EPA develops ethanol manual for producers." Bulk Transporter, November 19, 2007.  Accessed on August 19,
2009: http://bulktransporter.com/management/tank-truck/epa_ethanol_manual/.
7 "EPA ethanol Compliance Manual Available Online." ChemAlliance, September 5, 2008.  Accessed on August 19,
2009: http://www.chemalliance.org/news/news_detail.asp?StorylD=2272
8  "Reports."    Nebraska   Department  of   Environmental   Quality.    Accessed  on  August  19,  2009:
http://www.deq.state.ne.us/Publica.nsf/Pages/06-192
9  "Resources."    Nebraska   Ethanol  Board.     Accessed  on   September  10,   2009:   http://www.ne-
ethanol.org/resources/resources.htm
10 "BioFuel Considerations." Kentucky Department for Energy Development and Independence. Accessed on August
19, 2009: http://www.energy.ky.gov/dre3/renewable/considerations.htm
11 "Ethanol in Minnesota - information for industry."  Minnesota Pollution Control Agency.  Accessed on August 19,
2009: http://www.pca.state.mn.us/energy/fuels/ethanol-facilities.html

November 2009                                                                             Page 11

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                                                   Final Report
•   Level of Detail: Among respondents, there was disagreement over whether the document was too
    technical or not technical enough, but everyone interviewed and surveyed indicated that they found
    some use from the Manual.  In addition, most respondents thought the right amount of information
    was  available in  the  Manual, though a few indicated that too little  information was available and
    provided ideas for where content could be expanded or added.
•   Reference Guide: As noted  previously, almost all facilities who  responded to this evaluation had
    started  production prior to publication of this Manual, so they were familiar with the material; however,
    they expressed appreciation for having the information in one place to refer to when needed.
•   Contact Information:  Respondents  saw  the value of having specific state and EPA contacts in
    Appendix A of the Manual to  refer to for more information, but noted that it would be more difficult to
    keep up to date than other information in the Manual.

Interview  Findings
                                                                 "If  anything  you're trying  to
                                                                 reach  a broad audience, and
                                                                 the more broad you make it the
                                                                 less technical it becomes...  so
                                                                 it doesn't necessarily have the
                                                                 right level of detail for the finer
                                                                 points  in  regulation  for  all
                                                                 intended  audiences."  -  EPA
                                                                 contact
All  interviewees provided  an overall  positive  assessment of the
content and organization of the  Manual,  indicating that the Manual
was organized  well and was very  readable.  Comments  included
that it was  helpful that the Manual was organized by EPA program
areas and that the language was very straight forward.  Critiques of
the Manual from some respondents were that it may be slightly too
technical for those new to the field of environmental compliance and
permitting and that its length may prevent some  potential users from
perusing  it.  However,  no interviewees indicated that they saw
opportunities for shortening  the   Manual.    Other interviewees,
however, indicated the opposite—that  the Manual was too general
and needed more detail to be more useful to ethanol plants.  One state interviewee noted that when a
facility first  contacts them about an environmental  permit, their state agency has a multi-media group that
provides  information on the requirements for the facility;  gives  permit application forms and  any other
relevant handouts; and offers to have  an in-person meeting for any questions. This outreach provides
much of  the same information found in  the Manual, so the Manual  itself was less useful in providing
unique information to this state's facilities.
All but one  of the facilities interviewed had started  up production prior to publication of this Manual, so the
general content of the Manual was already  familiar to  them.  The one facility that  started up after the
Manual was published used it as a reference guide for construction and referred to it occasionally during
                                       operation.  Other respondents noted that the Manual overall
                                      was very useful, in that having all the information in  one place
                                      was in  itself helpful.   Another facility respondent  said their
                                      facility had put  together a  calendar based on the Manual to
                                       help track due dates for permits and plans.
  "There  are  differences  between the
  contacts  in the  appendix and the
  contacts on the Iowa DNR Web site.
  In some cases, the contact  section
  in the manual lists the head  of the
  department,   which    I   probably
  wouldn't  need to  contact.   Before
  using the contact  information in the
  manual I would check the Iowa DNR
  Web site to make sure  I contact the
  appropriate   person."    -  Ethanol
  facility
                                      The evaluators specifically asked all interviewees  about the
                                      "Who to Contact" section  of Appendix A and whether this
                                      section was useful and any ideas to improve it. Interviewees
                                      indicated  that the  contact information  in Appendix A  was
                                      helpful, though approximately half of the interviewees had not
                                      looked at this section until asked about  it during the interview.
                                      A common comment on Appendix A was that it might be  hard
                                      for  EPA to keep the  contact  information  current  and  that
                                      putting titles rather than contact names could help with this
November 2009
                                                                                        Page 12

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
challenge. Others, however, preferred to keep the specific contact names in the Appendix and have EPA
update the information frequently, perhaps on  a linked Web site.   One facility said  that the contact
information provided in the Appendix did  not always match the contact information found on their state
Web site or was not the appropriate level of staff person. One state interviewee said that he has received
calls from around the United States with questions about ethanol regulations and wished to have a list of
all state contacts for ethanol issues, rather than just those in Region 7.

Survey Findings
•   Survey respondents were generally positive about the Manual and its readability and organization.
    Twenty of the 21  respondents, or more than 95 percent,  who answered questions on readability
    indicated that the Manual was somewhat easy or very easy to read (see Figure below).
                             Satisfaction with Manual's Readability
                                                 N = 21
                                                   47.6%  47.6o/0
                                      0.0%__4.

    Sixteen (76 percent) of respondents said  it was easy or very easy to navigate to areas of interest,
    while five  (24 percent) said it  was  neither easy  nor difficult to navigate  the  Manual.  Twelve
    respondents (almost 60 percent) thought there was the right amount of information in the Manual (see
    Figure below).
                          Amount of Information in the Manual
                                             N=21

                                 Slightlytoo much
                                      5%
                                                                        Significantly too
                                                                          much (0%)


                                                                        Significantly too
                                                                          little (0%)


                                                                        No opinion (0%)
November 2009
     Page 13

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                                                  Final Report
Topic Area III: Changes in Understanding and Behavior
Has the existence and use of the Manual led to increased understanding and behavior change to support
improved compliance?

Overarching Findings
This evaluation indicates that  existence and  use of the  Manual has led to increased  or enhanced
understanding and knowledge for ethanol facilities. However, this evaluation cannot conclusively say that
the Manual has led to any significant changes that would affect compliance on the part of facilities.
•   Understanding and Awareness:    All  of the  facilities  and  contractors said the  Manual  was
    comprehensive and was helpful for increasing  or solidifying their understanding  of environmental
    compliance requirements.
•   Behavior Change:   With only a  few  exceptions, respondents were  not able to identify specific
    instances where  facilities  had  more awareness  of applicable federal regulations  or had made
    compliance or management changes as a result of the Manual.

Interview Findings
                                                         "I used it because we were brand new,
                                                         and it was great for starting  up. Now
                                                         that we're in  production I refer back to
                                                         the Manual. I also belong to a group of
                                                         ethanol  producers,  and many times
                                                         people will call me with questions.  I
                                                         use it as a  reference  tool to help
                                                         answer them." - Ethanol Facility
All of the facilities and  contractors  said the Manual was
comprehensive  and  was  at  least  somewhat  helpful for
increasing their understanding of environmental compliance
requirements, or solidifying what they  already knew.  The
contractors  interviewed  were  all very  familiar  with  the
applicable environmental rules, but all said that they thought
the Manual would be helpful to a facility starting out  in the
construction and  operation process.   State interviewees
noted that if the ethanol  industry rebounds, the Manual
would be very useful for helping new operators understand
requirements.
EPA and state compliance staff were asked whether facilities seemed more aware of applicable federal
environmental requirements as a result of the Manual. One  EPA interviewee noted that there had been
increased awareness of the Risk Management Program (Clean Air Act, Section 112 (r)); this interviewee
said that, prior to the Manual, many plants did not realize they were covered under this regulation. Other
compliance staff noted that there may be increased awareness among the regulated community, but that
                                    they could not conclusively point to the Manual as the cause of
                                    that knowledge.
  "It allowed us to not ship out work
  to consultants; we did all the work
  in house.  In the  past  we  would
  have just hired a  consultant, so I
  believe it  saved  us  money and
  effort." - Ethanol Facility
                                    Facilities, in general, could not point to instances where the use
                                    of the  Manual  had  influenced  a compliance  change.   One
                                    exception was a facility that had started up after the Manual was
                                    published, who said,  "Before we started up our plant, we read
                                    the Manual and realized that we had missed a permit," which
                                    prompted the facility  to secure the permit.  No facilities could
                                    point to any environmental or health outcomes that resulted from
the use of the Manual. One facility noted that the main impact on their management processes was that
the information in the Manual allowed the facility to do more of their permitting work in-house rather than
hiring outside consultants.
November 2009
                                                                                       Page 14

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
One EPA interviewee recalled being contacted because of someone reviewing the Manual and having a
follow-up question.

Survey Findings
Overall, survey respondents expressed satisfaction with the information  thoroughness of the Manual.
Sixteen respondents, or approximately 76 percent, were  satisfied or very satisfied with  how well the
Manual met their needs for understanding compliance information, and five,  approximately 24 percent,
were neither satisfied nor dissatisfied;  none expressed dissatisfaction.  One contractor respondent was
dissatisfied with the information thoroughness of the Manual, suggesting  that the air section should be
updated and expanded with more specific details.  All other respondents were either satisfied or very
satisfied with the information thoroughness (15 respondents,  or greater than 70 percent) or were neither
satisfied nor dissatisfied (five respondents, or 24 percent).

Compliance Data Analysis
         Table 1. Pre- and Post-Manual Publication Ethanol Violations
Informal Violation or
Notice
Pre-Nov. 2007
48.7%
Informal Violation or
Notice
Post -Nov. 2007
34.6%
Formal Violation
Pre-Nov. 2007
11.5%
Formal Violation
Post-Nov. 2007
9.0%
         N=78
Analysis  of EPA's Enforcement and Compliance  History Online (ECHO) database provided the above
information.  ECHO provides an overview of a facility's environmental record under the Clean Air Act
(CAA) Stationary Source Program, the Clean Water Act (CWA) National Pollutant Elimination Discharge
System  (NPDES), and the Resource  Conservation and Recovery Act (RCRA), as well as state and
federal responses.
This information shows that there  had been a decrease in informal  violations  since the Manual was
published in November 2007.  Prior to November 2007, 48.7 percent of the facilities in Region 7 had
received  an  informal violation, compared to 34.6 percent after publication of the Manual.   For formal
violations, 11.5  percent of Region 7 facilities received a formal violation before publication of the Manual,
compared to 9.0 percent  after publication.  However, this evaluation cannot  conclusively state that
environmental  compliance increased because  of this Manual.  The  data  in ECHO may have been
incomplete or lagging and may not have completely shown the enforcement actions taken in all of 2009,
and the evaluators did not have access to information on the rate at which inspections were conducted for
facilities in the Region on a yearly basis.  In addition, this evaluation had incomplete information about
whether  all  of the plants  analyzed were in full  operation before November 2007 and whether  they
continue to be in operation at the date of this evaluation. Further, past enforcement actions can influence
current compliance with regulations, as can other factors—such as learning from the mistakes of other
facilities,  a downturn or  upturn  in business, or changes in personnel with compliance  responsibilities.
Finally, this evaluation did not have information about whether all of these facilities received and reviewed
the Manual and therefore what, if any, effect it may have had on their compliance behavior.
The evaluators also  looked more specifically at the facilities that were interviewed for this evaluation and
that responded to the survey. Below are the findings of this analysis:
•   One facility had  more than 15 informal violations between 2005 and 2007, but only two  in 2008 and
    2009, all pertaining to air regulations.  The facility interviewee did not explicitly credit the  Manual with
    helping improve  compliance, but did note that  having the information in the Manual would have been
November 2009
     Page 15

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
    helpful when the facility was trying to be in compliance when just beginning operation, because the
    Manual lays out all the regulations and needs clearly.  The  interviewee particularly noted that the
    section on the CAA was helpful, as CAA requirements can be very confusing to facilities.
•   Another facility that responded to the survey had received 15 informal CAA violations between 2007
    and 2009.   In the survey, the respondent from this facility expressed satisfaction with the CAA
    information  in  both  the building  and operation  sections  of the Manual.   This respondent also
    suggested that voluntary EPA inspections would be  a helpful way  to  learn  about  compliance
    information.  This facility indicated that they had received the Manual from a state contact, though did
    not indicate a timeframe for this interaction.
•   Another facility that  responded to  the survey  had received a formal enforcement action under the
    CAA earlier in 2009. This respondent expressed dissatisfaction with  the information thoroughness of
    the Manual and requested more specific examples and details.
As  noted above, this evaluation cannot conclusively  state that there  is any causal link between these
compliance actions and use of the Manual.  During discussions with the Region 7 Biofuels Team about
the findings of this report, members of the  Biofuels Team  noted  that more  consistent tracking of
compliance and beyond compliance  behavior and  targeting  assistance based  on more thorough
compliance information may be a path forward for this sector.
Topic Area IV: Improving the Manual
How could the Manual be improved to provide effective compliance assistance to ethanol facilities?

Overarching Findings
Respondents to this evaluation had a number of suggestions for how future versions of Manual could be
improved. The suggestions centered around additional  content, such as a section on cellulosic ethanol,
or around functional content, such as checklists, case studies, call-out boxes, and examples.

Interview Findings
Most interviewees had some general comments and suggestions for how the Manual could be improved
to provide effective compliance assistance, as well as specific suggestions for additions. They offered the
following suggested changes and additions to the content:
•   A few contractors requested more examples be provided in the Manual to define terms, such as what
    constitutes commencing construction and modifying a plant.
•   A contractor suggested including information on how long the permitting process takes, as well as a
    general discussion  of what permits would likely be required, as many operators start the process
    without any knowledge of the expected permit timeframes.
•   A contractor suggested  providing embedded hyperlinks to  the Code of Federal Regulations (CFR)
    Web site so that readers can look up the referring statutes.
•   A  contractor  suggested moving  the  Renewable  Identification  Numbers  (RINs)  requirements
    discussion to a later section of the Manual, rather than up-front, since this requirement comes further
    along in the facility operation process.
•   A contractor noted that some of the permits listed in the Manual did not seem applicable, and three
    contractors noted other  EPA permits and reporting  plans applicable to ethanol plants that were not
November 2009                                                                          Page 16

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
    mentioned in the Manual, including: Toxic Substance Control Act (TSCA) Form U Reporting and New
    sources performance standards (Subpart VVa) (released after the Manual was initially published).
•   A contractor disagreed with the statement found in the Manual that "all ethanol plants will probably
    generate  some  quantities  of hazardous  waste" (page 2-20), and noted  that some facilities were
    focusing on making use of their byproduct waste streams to make value-added products.
•   A facility suggested adding an executive summary to the front of the Manual to make the document
    more approachable to potential readers.
In addition, the evaluators specifically asked interviewees about any suggested changes to the "Who to
Contact" section of Appendix A.  There was no clear consensus from interviewees on whether this section
should provide direct numbers and names to contact people or to general support numbers. Interviewees
noted the benefit of having these specific names in  knowing whom to contact but noted that staff turnover
could cause this section to be out of date sooner than other parts of the Manual.  One state  interviewee
suggested expanding this section to include all applicable ethanol offices in state agencies so that those
outside of Region 7 could use the Manual.
Interviewees  were asked whether there were other areas of information that EPA should consider
covering in  future versions of the Manual. The following were the suggestions provided for this area:
•   An EPA interviewee noted that issues have arisen in  recent years around emissions from  fermentors,
    biochemical oxygen demand loading and  storm water ponds, and leak detection.   These issues and
    others that have come to the attention of regulators may  be candidates for specific discussions in
    future versions of the Manual.
•   An  EPA  interviewee  suggested that checklists would be helpful to  facilities that  have been in
    operation and were familiar with much of the Manual content.
•   A state interviewee noted that the Manual  was geared toward  building a  new plant,  but that any future
    version could include information  about requirements when making substantial modifications to that
    plant or process (e.g.,  using a new feed  stock, burning fiber).  Similarly, two contractors noted that
    ethanol plants were looking for ways to  integrate value-added products to their  production stream
    (e.g., oil extraction), and that including a discussion of requirements that plants should consider would
    be useful.  Related to this discussion, a state compliance  staff member suggested a section on
    byproduct handling.
•   An  EPA  interviewee suggested adding  information about state laws to make  the Manual more
    comprehensive,  while noting that it would be more difficult  to update that information.  Facilities and
    contractors, similarly, said that ethanol plants were responsible for a number of federal requirements
    to  different agencies,  including  the U.S. Department of  Transportation,  the U.S. Department of
    Homeland Security, the U.S. Army Corps  of Engineers, and the Alcohol and Tobacco Tax and Trade
    Bureau.
•   EPA and state  staff as well as contractors noted that future versions could include sections on
    cellulosic and other emerging types of ethanol.
•   State staff and contractors suggested that a future version of the Manual could include information on
    what should happen with federal permits when plants halt  production for any length of time.  In
    addition,  unless there is  renewed  emphasis  on  alternative fuels, there  may  not  be  as  much
    construction of facilities  in  the future; any future  versions  could focus  on operation  rather than
    construction of plants.
November 2009                                                                           Page 17

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
•   An EPA staff  member suggested  information  on community and  emergency response could be
    valuable for this audience and that EPA should incorporate this information or  link  to  another
    organization that has resources in this area.

Survey Findings
The survey asked facilities and contractors what things EPA should consider to improve satisfaction with
the Manual as a compliance assistance tool. The following were the suggestions provided in the survey
for additional content or functionality:
•   More examples and specific details throughout the text
•   A list of ethanol relevant resources, such as ethanol trade associations
•   A list of contractors or consultants in particular states and the services they provide
•   Checklists and "at a glance" materials and pages
In addition, a facility expressed dissatisfaction with the transition between the construction and operation
sections of the Manual, but did not elaborate on this response.  Another noted that the section on air
quality regulations needs updating and improving, but also did not expand on this comment.
The survey asked  participants how frequently EPA should update the Manual to be helpful to facilities.
Ten respondents (48 percent) suggested updating the Manual every three years.  The three respondents
who chose "Other" wrote in  that Region  7  should  update the  Manual whenever there were  regulatory
changes (see Figure below).
                       Which update frequency would be the most useful for the
                                            Manual?
                                              N=21
                          Other timeframe
                               14%
            Every five years
                 9%
                                                                       I don't intend to
                                                                         refer to the
                                                                        Manual again
                                                                            (0%)
November 2009
     Page 18

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report


Topic Area V: Other Compliance Assistance

What approaches other than the Manual would provide effective compliance assistance to the ethanol
facilities?

Overarching Findings
As  previously discussed, evaluation participants thought the Manual was a useful  tool  for conveying
compliance  information.   However, evaluation participants also considered other venues,  such as
conferences, EPA workshops, a more  interactive Web site, and one-on-one  interaction with ethanol
facilities to be additional valuable approaches for providing compliance assistance.

Interview Findings
During the interviews, facilities and contractors  were asked about other tools and approaches that EPA
could use to present the information that is  available  in  the Manual.   Five of the  ethanol  facility
representatives  noted that they had attended EPA workshops in the past and found them to be useful
sources of information about regulations and implementing them at their facilities.  In addition, a  few of
the contractors and facilities regularly attended conferences; additional or continued EPA presence would
be welcome  there to share information about new resources or requirements. In addition, interviewees of
all categories noted that having an up-to-date Web site that is easy to navigate was a very effective way
to convey information. One contractor suggested having an ethanol-specific Web page, either nationally
or regionally, to  provide links to resources and updates  on regulations.

Survey Findings
The survey asked  respondents about what other tools and approaches would be  helpful for conveying
compliance assistance information. Most noted that compliance assistance had so far been sufficient.
Below were the  other suggestions from the survey:
•   An email newsletter or listserv with reminders, updates, industry  contacts, suggested best practices,
    and regulatory changes applicable to the industry.
•   Voluntary EPA inspections without fines.
•   Additional workshops and meetings with open discussions between regulators and ethanol plants.
November 2009                                                                           Page 19

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
IV.   Recommendations

This evaluation assessed the effectiveness of an ethanol compliance assistance Manual developed  by
EPA Region 7.  The  evaluation  focused  on a  set  of overarching  evaluation  questions to inform this
assessment.  Evaluation participants clearly conveyed that the Manual was useful and valuable to  its
intended  audiences, and that it effectively communicates relevant information  for ethanol  plants.  The
following  recommendations focus on ways to improve the Manual for any potential future versions and
other ways that Region 7 could make this information available to its intended audiences.
The evaluators understand that there may be limited resources  available for updating the Manual and
limited time available for staff to provide other kinds of compliance assistance to ethanol facilities.  The
evaluators also recognize that the market for ethanol has been shifting dramatically, and that the demand
for new facility permits has not  returned to the peak it was in when  Region 7 first released the Manual.
These and other related factors  will likely influence demand for the Manual and feasibility of implementing
any of the evaluation's recommendations.  The below recommendations are organized thematically and
not by priority.

1. As a  first step, consider whether to tailor the Manual's content and distribution to the national
    level.
The Manual is available  to the  public on the Region 7 Web site, and as such  it is available nationally.
However, the contact information  in Appendix A  is focused on Region 7 only, and EPA's national offices
have not  posted the Manual on their Web  sites. EPA and state  compliance staff and contractors noted
that the Manual's  content was not specific to Region 7, and would be of interest to other Regions with
significant ethanol industries.  Already, some  state agencies such  as the Kentucky Department for Energy
Development and  Independence and the Minnesota  Pollution Control Agency have linked to the Manual
on their Web sites, noting that it may be of interest to their states' ethanol facilities.
The Region 7 Biofuels Team noted that they  had written the Manual  in such a way that it could be easily
adapted to other EPA  Regions.  The Region  7-specific contact information, for example, was included in
an appendix to enable easy substitutions with other Regions' contact information. Sending the Manual to
EPA staff in other Regions for posting and distribution would  be a simple first step to making this
information more widely available, and Regional staff or Headquarters could take the lead on  revising the
contact Appendix  and  any other sections that needed another Region's specific information.  Staff in the
other Regions and at state environmental agencies would likely need to be available to answer questions
about the information  provided  by the Manual, which would require  a commitment of staff time in other
Regions with significant numbers of ethanol plants.
The evaluators recognize that there may  not be a  clear national EPA office or program to take  over
responsibility for the current Manual, any updates to the Manual,  or any coordination between or  among
Regions for the Manual.  Similarly, other Regions may not have an equivalent biofuels team or group to
coordinate the  Manual's distribution or incorporate regional contact information.  For these reasons, if the
Manual were to be used  nationally or in other Regions,  it may be necessary for the Biofuels Team in
Region 7 to identify individual  "champions"  for the Manual's  use and  updates at the national  and/or
regional  levels.   Some  level  of ongoing coordination  between Regions  or  between  Regions  and
Headquarters may be needed.
November 2009                                                                          Page 20

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                         Final Report


2.  Consider additional outreach strategies and targeted audiences.
Region 7 undertook a number of activities to advertise the Manual's existence, including a news release,
providing postcards at conferences, emailing those who had expressed interest, and highlighting the
Manual  during media interviews.   The Region could continue to advertise the Manual and any future
versions with these mechanisms, but could also expand its outreach to consider additional audiences and
outreach strategies.
The evaluators  recognize  that  Region  7 had  limited  resources to conduct  Manual  outreach  and
advertisement.   Allocating  at  least some resources to  outreach would allow the Region  to  conduct
activities such as advertising in trade journals  relevant to the ethanol industry.  However, Region 7 could
also undertake activities that would not require significant resource investment and would be more likely
to reach the targeted audiences relatively quickly:
•   Region  7 could develop and maintain an ethanol-industry  email  listserv that regularly sends out
    updates and news to subscribers.  Region 7 could allow anyone to join or have a moderated listserv
    of only  ethanol plant managers.  This approach would be a simple way to keep in contact  with
    interested stakeholders  and alert them to any future versions or updated Manual, or to remind them of
    the Manual's existence,  as well as send out other relevant information.
•   When Region 7 staff members visit ethanol plants on site visits, the staff could bring an electronic
    copy of the Manual or could provide a  postcard with the Web link to the Manual.
•   If the Manual is updated with new information, Region 7 could gather a small focus group of three to
    five  ethanol facility contacts by phone  or in-person.  This group could discuss the best ways to inform
    ethanol  plants about any changes,  and whether the  industry needs a fully updated Manual with the
    information, or whether an email or Web  site with the changes is sufficient.  This approach would
    allow Region 7 to  learn  about any venues  (e.g., upcoming conferences) that could be key distribution
    points, and would  also allow Region 7  to target its resources strategically.
While ethanol plants were  the primary targeted audience for this Manual, ethanol-industry  contractors
could be an effective audience for future  outreach. Many of the facilities interviewed said that they use
contractors for much of their environmental compliance and permitting work, and rely on these contractors
for updates and information on regulation.  As  there are a smaller number of contractors and  consultants
working  with existing and potential  ethanol facilities,  making  the   Manual specifically available  to
contractors  could more directly  reach an  important audience and perhaps  ultimately reach  a wider
audience.   In addition,  having contractors review draft  Manual updates may be  a  productive way to
ensure that the Manual is helpful to the target audience.
3.  Consider making this tool primarily Web-based, with a hard copy  Manual as a supplement.
Nearly all  evaluation  participants  accessed the Manual  on  the Internet,  and  many suggested  that the
Manual  include  more Web-based content, such  as updates and additional or more current contact
information.   Transferring  the Manual to a  current,  Web-based  version would  require some effort,
particularly if user interactivity such as topic "drill  downs" and links to related topics or contacts, were to
be incorporated.  Making the content Web-based,  however, would make the Manual more accessible and
useful to many users and  likely easier to maintain and update  by EPA.  In addition, Internet search
engines could find specific  sections more easily,  and Region 7 could implement Web feeds that would
allow users to subscribe and be notified of any changes.12  A version formatted for printing could still be
12 "Web feeds" are a format for providing users with frequently updated content. Content distributors (in this case,
Region 7) develop a "feed" and allow users to subscribe to it. Users are alerted through a "feed reader" or through

November 2009                                                                            Page 21

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                         Final Report
available on the Web for readers who wanted to download it for their office reference library or for EPA
and state compliance staff or others who would want to take hard copies to facilities they visit.
The below are some ways that Region 7 could implement this recommendation:
•   Region 7 could simply upload the content onto a series of HTML Web pages.  This approach would
    allow EPA staff to make changes quickly to  the content  of the Manual, without republishing the
    document entirely.  Examples of this approach at EPA include the EPA Lean in Government Starter
    Guide13 and the EPA Guide for Industrial Waste Management.14
•   Region 7 could upload the Manual to the Web  in a wiki-format,15 which would allow EPA staff to edit
    the content even more easily than HTML pages, and could allow other non-EPA users to edit the
    content. This ease of updating can help ensure that content is up-to-date and relevant to its intended
    audiences. Region 7 could upload the content to an EPA-hosted or third-party hosted wiki and then
    designate  through  assigned  log-ins which groups  (e.g., EPA staff, state compliance staff, facility
    contacts) are allowed to contribute edits to the document and create discussions about specific pages
    or sections.  This  approach could allow a greater cross-section of stakeholders to contribute to
    improving  the Manual, and could result in  a more  valuable resource for ethanol plants looking for
    compliance assistance information.  If Region  7 decides to go this route to make the Manual Web-
    based,  they can look to the lessons learned of  similar efforts at EPA and other federal agencies.  For
    example, EPA's Watershed Central Wiki is a Web site based on the MediaWiki platform that is open
    to EPA employees and identified partners to  provide information on watershed resource management
    best practices, organizations, and plans.16
•   Region 7 could develop a Web site that displays the content of the Manual on a series of HTML or
    Wiki Web  pages (discussed above), and also  links  to more dynamic content and resources internal
    and external to  EPA, such as upcoming meetings or relevant conferences.  For example, the EPA
    Office  of  Wastewater Management  has developed  a Web  site  on  NPDES  (National  Pollutant
    Discharge Elimination System) permits.17   It  links  not only to compliance information, but also to
    relevant new publications and training materials, discussions of current issues, and links to upcoming
    meetings and conferences.  While incorporating and updating this material would require more staff
    time and resources on a continuous basis, it could become a frequently used and valuable resource
    for ethanol plants and their environmental managers.
4.  Update with current information on regulations.
If a future version of the Manual is published, it could incorporate the latest information on applicable
federal environmental regulations, permits, plans, and approvals for ethanol  plants.   Alternatively,  a
stand-alone supplement to the  Manual that focuses solely on recent changes and updates  could be
created.  If the Manual was Web-based (see Recommendation 3), regular maintenance would ensure that
all of these components are kept current.  If the Manual continued to be hard-copy based, then having
their email when new content is available.  An example of this is EPA's Office of Public Affairs news release pages,
which alerts users whenever a new press release is issued; see http://www.epa.gov/newsroom/rssfeeds.htm.
13See http://www.epa.gov/lean/starterkit/index.htm
14See http://www.epa.gov/waste/nonhaz/industrial/guide/index.htm
15 A wiki  is a Web site  that  allows easy creation and editing  of content for users.  Wikis are  often  used for
collaborative editing of a document that has content that changes frequently and/or could  benefit from multiple
perspectives.  Wikipedia (http://www.wikipedia.org) is one of the best known wikis.
16 See  https://wiki.epa.gov/watershed/index.php/Main_Page; Access is limited to  EPA employees and identified
partners.
17 See http://cfpub.epa.gov/npdes/index.cfm

November 2009                                                                            Page 22

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
links within the Manual to an EPA Web page that provides real-time updates to the Manual content would
keep the information current for ethanol facilities and contractors looking for updates.
5.  Create an executive summary to encapsulate the main points of the Manual.
A few participants in this evaluation noted that the length of the  Manual could detract potential readers
from reviewing the Manual; however, no participants noted opportunities for shortening the Manual.18  In
addition, several participants thought that a short executive summary or equivalent synopsis would give
readers an overview of the content and any potential sections of interest without taking away any of the
content within the Manual.  There is currently a front section ("What is the purpose of this manual?") that
provides two paragraphs on the purpose of the Manual and some disclaimers on what the Manual does
not cover; however, this section is  not an executive summary in the way that readers expect.  Similarly,
the Introduction  covers three topics  (requirements for  renewable fuels,  the  renewable fuel standard
program, and a brief overview of the ethanol production  process), but does not include a list of the laws
that apply when constructing, modifying, or operating ethanol plants or any related summary information.
Creating a new executive summary or overview section to summarize the main topics of  each of the
chapters could be one way to  meet the stated need  of respondents.
6.  Incorporate more "at a glance" information.
Expanded "at a glance" information, such as lists  of resources,  call-out boxes, checklists (e.g., on the
steps to take), and calendars can be integrated throughout the Manual to highlight particularly important
information. Respondents to this evaluation indicated that they would like  more of this kind of content so
that they can quickly refer to  information of interest.  The evaluators recognized that the Biofuels Team
intentionally avoided creating new guidance with the Manual, and as such that there may be reason  not to
create  action or requirements  lists  that could constitute guidance.  If this has not been done already, it
may be worth exploring the feasibility of drawing from existing guidances to develop "how to" checklists or
other short reference lists either at the regulation, media, and/or cross-media levels.
Similarly,  a  generic schedule  that a plant could modify and download could  be very useful.  Such a
schedule could, for example, indicate a starting date and include the deadlines  (e.g., date +  90 days) for
the related  required procedures, submissions, and other information.  It may be  that plants or their
contractors already have these tools in paper or electronic format and may be willing to share them for
use by  EPA for this purpose.  As  mentioned earlier in this report, one facility interviewee developed a
calendar for this purpose after reading the Manual.
Several participants raised the idea of listing resources beyond the contacts already included in Appendix
A.  Some said that a list of contractors would be  of interest, though they also said that caution may need
to be exercised in doing so to avoid perceived or  actual conflicts of interest.  Some thought that there
were precedents for listing contractors and that there would  be no problems in doing so as long as there
were multiple contractors listed.  Others thought that directing parties to external  Web sites  that already
list  contractors would  be a relatively easy way  to achieve this change, though it is not clear  that there is
such a resource.  Other resources that Region 7 could consider could be  lists  of facilities that would be
willing to discuss  their experience with others and  contacts  from  other regulatory agencies,  such as the
Occupational Safety and Health Administration  (OSHA).
18 One way to reduce the manual's size would be to reformat it with fewer pictures (though the pictures make it a
much  more visually pleasing document), reduce the large margins, and modify the spacing.  Converting the Manual
to a Web-based format may or may not assist with length depending on how the Web site is designed.  This would
not serve as a substitute for including an executive summary but would likely help with the initial impression voiced by
several participants that the  Manual seems very long.

November 2009                                                                             Page 23

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
In developing the Manual, the Biofuels Team has considered about whether a "flip book" of key issues
and important points would be feasible.  The evaluators think that anything along these lines, regardless
of the format (e.g., flip book, standard paper format, electronic spreadsheet) would be useful for readers.
7.  Provide more specific examples and case studies.
Participants commented that the information in the Manual was sometimes vague or too general and that
specific examples  would  be  helpful  for  understanding.   For example,  providing  examples of  what
constitutes a modification to a  building would  be helpful to a reader wanting to understand what is
applicable to their building. More narrative discussion or case studies,  either real or hypothetical, can
also help a reader relate their facility to the regulations discussed within the Manual.
8.  Expand content  to  include regulatory  information about  topics of interest  to  the  target
    audience, such as cellulosic ethanol production and  byproduct diversion into other products.
There were  many suggestions for specific content to add  to  the Manual for future versions (see the
findings on Topic Area IV: Improving the Manual). Cellulosic ethanol production and byproduct diversion
into other products were  mentioned most often by the state compliance staff, facilities, and contractors
who participated in the evaluation.  Cellulosic ethanol production is expected to increase as producers
look to different feedstocks to take advantage of new incentives, research findings and changing  market
conditions, and facilities  are interested in the  different  federal environmental regulations that may be
applicable. In addition, during this evaluation, ethanol facilities indicated that they were looking for ways
to take  the  byproducts of ethanol production,  make them into value-added  products,  increase  profit
margins, and decrease waste disposal.  Facilities may find  value  in a  discussion  in  the Manual of
regulations and  requirements to consider when changing  operations in these ways.
9.  Expand information  on beyond compliance efforts and best practices.
Although the Manual is focused on current federal environmental regulations and rules, a section on how
a facility could go beyond compliance and implement best practices may be of interest to  some facilities.
In particular, newer or  smaller facilities may not  know  about  best  practices that  could provide
environmental, safety  and/or economic benefits; having this information in one place could  be  useful.
Currently, the Manual has a section in the  construction chapter about the Pollution Prevention Action and
refers to the EPA Pollution Prevention Web site.19  Region 7 could evaluate  whether this is sufficient
information or whether expanding this section with additional best practices  could be a way to incorporate
more  beyond  compliance information  for the audience.   The evaluators  recognize,  however, that
providing  beyond compliance  information is   a secondary priority  following  the Manual's focus on
compliance assistance.
10. Apply the  lessons from this evaluation to the  Region's other  compliance assistance and
    beyond compliance efforts.
The evaluators understand that Region 7's compliance assistance activities are largely integrated into its
broader compliance-enforcement program. The Region  has a compliance assistance coordinator who
serves the regional media program staff as a  policy disseminator and a data quality control check for
measuring activities. The technical and enforcement staff in the other regional programs incorporate the
actual compliance assistance activities into their other routine actions. More broadly, the Region focuses
on conducting  inspections, following up  on  potential violations, and, in general, working within the
traditional compliance-enforcement framework that has been the hallmark of EPA's compliance approach
for over 30 years. Region 7, like other Regions, also oversees the  compliance enforcement work of the
1Q
  See http://www.epa.gov/p2/


November 2009                                                                           Page 24

-------
  Evaluation of the Effectiveness of an Ethanol Compliance Manual                         Final Report


  Region  7 states and  reports  back to EPA  Headquarters Office of Enforcement and Compliance
  Assurance (OECA)  and on its activities and progress through routine  reporting channels.  Compliance
  assistance  is,  in other words,  primarily  an  "add  on" to  more  "core"  programmatic activities and
  responsibilities.
  Within  this  broader construct, the Region conducts  both  ad-hoc and  routine  periodic compliance
  assistance work.  The Manual is an ad-hoc example that the  Region took on after a dramatic shift in the
  number of new permit requests for ethanol facilities as explained  in this report.  In another  instance,
  Region 7 is focusing on improving compliance with a new lead (Pb)  rule, which was promulgated in April
  2008. This rule requires workers to follow lead-safe work practice standards to reduce potential exposure
  to dangerous levels of lead during renovation and repair activities.20 The Region  has been conducting
  outreach with thousands of contractors and citizens who renovate, repair,  or repaint older buildings that
  may have lead-based  paint to support compliance  with the new rule.  The Region  also discusses
  compliance assistance  activities each quarter with the Region 7 states.  At  these meetings, the state and
  regional  staff  take  turns sharing their  compliance  assistance  activities  among  discussions  of other
  activities for collaboration.
  Within this context,  the evaluators, in consultation with  Region 7  staff,  have identified some options for
  applying  the lessons learned from the evaluation of the ethanol  compliance assistance manual to the
  Region's broader compliance assistance activities.   These options, which can serve as the  basis for
  further discussion within the Region, are as follows:
 i.    Determine whether the Region would  like  to initiate more compliance assistance activities like the
      ethanol  compliance assistance manual that are proactive  in  nature and intended  to help  raise
      awareness amongst a targeted regulated community that has been identified  as  needed additional
      support. Doing  so would require additional planning and coordination and therefore the "blessing" of
      senior managers who would deploy staff resources for this purpose.  In addition, there may need to
      be further exploration  of the  relationship between  the compliance assistance work and the more
      customary  interactions  that compliance inspectors  have with  the regulated  community to clarify
      respective roles and responsibilities.
 ii.    Consider options for cross-media compliance assistance integration like the Biofuels' Team effort on
      the  ethanol manual.   Evaluation participants  expressed their appreciation  for having all  major
      regulations described in one place instead of media-by-media and program-by-program.
iii.    Consider additional opportunities for coordinating with, as well as learning from,  the Region 7  state
      compliance programs.  Although Region 7 already works hand-in-hand with the state compliance
      enforcement programs, there are  likely opportunities to  be more  proactive in terms of identifying
      potential compliance gaps or assistance opportunities, learning about planned or ongoing compliance
      activities that the states are conducting and coordinating  on compliance assistance implementation.
      There are  likely efficiencies and economies of scale for doing so; for instance, in  providing  regional
      compliance assistance materials (such  as the Manual) to  state compliance staff and vice versa, or in
      having  strategic planning  meetings intended to design and coordinate  Region-wide compliance
      activities for the upcoming  years (this would be materially different  from the current update-oriented
      quarterly meetings).
iv.    Take advantage of Internet-based tools and options.  Tools such  as  Webinars, interactive guides,
      Wikis, and topical  drill-down Web sites (such as the one discussed  in the context  of the ethanol
  20 See: http://www.epa.gov/region7/citizens/lead.htm


  November 2009                                                                            Page 25

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                        Final Report
    compliance assistance manual) provide opportunities for getting compliance information out to a wide
    range of people quickly.
In general,  these ideas and options will depend on whether Region 7 wishes to invest in  more targeted
compliance assistance on top of the  current compliance-enforcement activities.  There are likely to be
needs related to existing or new rules where additional compliance assistance focus would help to raise
awareness  and support compliance before any inspectors contact regulated entities.  Perhaps Region 7
is already exploring such options in relation to other sector outreach, or perhaps this is an  area that is
worth exploring.
As a first step, Region 7 may wish to identify a point person or small group of individuals to further explore
the  options for either following up on recommendations related to the ethanol compliance assistance
manual itself, or on taking a next step on additional targeted compliance assistance that can  benefit from
the  lessons gained from the Manual evaluation. It is likely that a targeted scoping exercise  could result in
a short set  of viable  options and recommendations for future "value added" compliance assistance that
Region  7,  its  states,  and  the  Regional regulated  community could  all utilize to  support improved
understanding of and compliance with the law.
November 2009                                                                           Page 26

-------
?/ERA
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
 November 2009
EPA-100-R-10-001
        Evaluation of the
        Effectiveness of an
        Ethanol Compliance
        Manual
        Report Appendices

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
              Table of Contents









              Appendix A.  Customer Satisfaction Survey	1



              Appendix B.  Customer Satisfaction Survey Results	10



              Appendix C.  Interview Guide for Ethanol Facilities	19



              Appendix D.  Interview Guide for Ethanol Contractors	22



              Appendix E.  Interview Guide for State Compliance Staff	24



              Appendix F.  Interview Guide for EPA Staff	26

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
    Report Appendices
         EPAICRNo. 1711.05
     OMB Control No. 2090-0019
   EXPIRATION DATE: 8/31/2009
Appendix A.   Customer Satisfaction Survey

      COMPLIANCE ASSISTANCE MANUAL

                 FOR  ETHANOL  FACILITIES
                 Survey of Customer  Satisfaction
      Thank you for participating in this survey so that EPA can serve you more effectively.

  Assuming you have already read the compliance assistance manual, Environmental Laws Applicable to
  Construction and Operation of Ethanol Plants, we anticipate that the main part of this survey will take
  approximately 10-20 minutes and the optional questions at the end will take approximately an additional
  10 minutes. Your participation will help EPA improve ethanol facilities' satisfaction with this manual.

  You may also take this survey on-line at the following website:
  http://www.survevmonkey.eom/s.aspx?sm=2uWpdtMi8JSWiH 2bkMbQ35g 3d 3d.
  Submitting your responses online will take the same amount of time  and will help us to analyze the
  results and provide feedback to  EPA more quickly.
  If you choose to fill out the paper survey instead, please return
  your responses by July 10, 2009 in the enclosed self-addressed
  stamped envelope.

  Note about respondent confidentiality:  The contractors evaluating
  the ethanol facilities' and contractors' satisfaction with the compliance
  assistance manual, Environmental Laws Applicable to Construction and
  Operation of Ethanol Plants, are gathering basic information to better
  understand how useful different types of customers have found the
  Manual to be. Informational responses will be generalized and made
  anonymous for the final results and no facility or contractor names or
  other identifiers will be provided to EPA as part of the results, unless a
  facility or contractor explicitly indicates at the end of this survey that
  they would like their responses to be identified.
    '-.'• I ;:M|, M | M ',( LAWS
    Ethanol Plants
    T»       m-a-CKH.'!
•jV  .—\  ±^!L
      .'->,%
             •
  Industrial Economics, Incorporated (http://vwvw.indecon.com/) and Ross & Associates Environmental Consulting,
  Ltd. (http://www.ross-assoc.com) are the contractor team. For more information, please contact Anna Williams
  (anna.williams@.ross-assoc.com) or Amy Wheeless (amv.wheeless@.ross-assoc.com) by email or at (206) 447-
  1805.
November 2009
              Page 1

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                     Report Appendices
I.   PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER
    1.  Prior to receiving this questionnaire, were you aware that EPA developed the Compliance
       Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available?
       o
       Yes
O
No
   2.  Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities,
       published by EPA in November 2007?  If No, please skip to Question 17.
       O
       Yes
O
No
   3.  Please check the method below that indicates how you learned about the Manual
O
Postcard
distributed from
EPA
0
Meetings /
conferences (e.g.,
O
EPA news release
0
Other
correspondence
O
EPA radio or TV
interview
0
Other method
(please specify)
O O
EPA Web site State contact
0
Don't know/don't
remember
     FEW workshop,    from EPA
     CenSARA's National
     Environmental
     Biofuels
     Conference)
     If other:
   4.  Why were you interested in receiving the Manual? (check all that apply)
     00                        00
     Wanted  to  learn  more Wanted  to   learn   about General reference   Other
     about applicable Federal applicable Federal laws for
     laws  for construction  of operation of facilities
     facilities
     If other:
November 2009
                                                                Page 2

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                    Report Appendices
        Optional response: O I received the Manual without asking for it (check here if applicable)
    PART 2: ACCESSING THE MANUAL
    5.  Please check the method below that indicates how you accessed the Manual.
     O
O
O
O
O
     Downloaded the    Received a        Received a CD
     Manual from the    Compact Disc (CD) from a state
     EPA Web site       from EPA         contact
                                   Received a CD      Received a hard
                                   from another      copy from EPA
                                   contact outside of
                                   my facility
     O
O
O
     Received a hard    Received a hard   Other method
     copy from a state   copy from a       (please specify)
     contact            different (not
                       state/EPA) contact
                       outside of my
                       facility
     If other:
    6.  Please rate how satisfied you were with the ease of accessing the Manual.
     000000
     Very           Dissatisfied     Neither        Satisfied        Very Satisfied   No Opinion
     Dissatisfied                    Dissatisfied nor
                                   Satisfied

     Please describe any ways you think EPA could improve access to the Manual (optional):
    7.  Have you read any part of the Manual?  If No, please skip to Question 17. If Yes, please continue
       to Part 3 of the survey.

       O            O
       Yes           No
November 2009
                                                               Page 3

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                      Report Appendices
    PART 3: STRUCTURE AND READABILITY OF THE MANUAL
    8.  Please rate your satisfaction with the Manual's readability.
     0000
     Very difficult to  Difficult to read Neither easy
     read                         nor difficult to
                                  read
                           o
             Somewhat easy Very easy to
             to read        read
                         o
                         No opinion
    9.  Please rate the ease of navigating through the Manual to get to areas of interest.
0
Very difficult to
navigate to
areas of interest

0
Difficult to
navigate to
areas of
interest

10. How would characterize the
O
Significantly too
little
O
Slightly too
little
0
Not easy, but
not difficult to
navigate to
areas of
interest
0
Easy to
navigate to
areas of
interest

amount of information presented
O
Right Amount
O
Slightly too
much
0
Very easy to
navigate to
areas of
interest

in the Manual?
O
Significantly
too much
0
No opinion


O
No opinion
    We welcome any comments on your response:
IV.  PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL
    11. a. Please rate your satisfaction with how well the Manual met your needs for understanding
       compliance information.

     Very Dissatisfied  Dissatisfied
     O              O
Neither
Dissatisfied nor
Satisfied
O
Satisfied
O
Very
Satisfied
O
No Opinion
O
November 2009
                                                 Page 4

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
       b. Please rate your satisfaction with how clear the information provided in the Manual was on
       where to go for more information.
                                     Neither
                                     Dissatisfied nor                 Very
     Very Dissatisfied Dissatisfied      Satisfied         Satisfied      Satisfied     No Opinion

     000              000

       c. Please rate your satisfaction with the information thoroughness of the Manual.
                                     Neither
                                     Dissatisfied nor                 Very
     Very Dissatisfied Dissatisfied      Satisfied         Satisfied      Satisfied     No Opinion

     o              o             o              o           o          o
    We welcome any comments on your response:
    Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If so,
    we welcome any explanation and suggestions for improving your satisfaction with that section or
    chapter below.

    OYes     ONO
    12. What else should EPA consider to improve your satisfaction with the Manual as a compliance
       assistance tool?
November 2009                                                                          Page 5

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                    Report Appendices
    13. Please rate your overall satisfaction with the Manual as a compliance assistance tool.
     Very Dissatisfied  Dissatisfied
     o              o
              Neither
              Dissatisfied nor
              Satisfied

              O
                           Very
             Satisfied       Satisfied      No Opinion
             O           O           O
V.  PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE
    14. If EPA were to update the Manual, which update frequency would be the most useful?
     O
     Annually
     If other:
O
Every 3 years
O
Every five years
O
Other timeframe
(please specify)
O
I don't intend to
refer to the
Manual again
    15. What other tools and approaches would you prefer for sharing compliance assistance
       information concerning ethanol facilities?
VI. FINAL QUESTIONS!
    16. What category best describes your position?

     O                      O
     Facility manager          Facility compliance officer
                                 O
                                  o
                                Other   facility  role Contractor
                                (please specify)
     If other:
    17. Can we identify your facility or company (by name) as having participated in this survey?

       OYes         ONO
November 2009
                                                                Page 6

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices
       Facility or Company Name (optional)
PRIMARY POINT PERSON (OPTIONAL)

If you are willing to provide the name of a primary point person who could be contacted by the
contractors with any follow up questions or correspondence, please list that person here.
Name:
Phone Number:
E-mail Address:
                          Thank you for participating in this survey!

       Please return your responses using the enclosed self-addressed stamped envelope by
                                      July 10, 2009.
November 2009                                                                      Page 7

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices
Optional Section: if you would like to provide more specific feedback on each Section of the
Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer
the following questions.


   1.  Please rank your overall satisfaction with the ease of understanding, information thoroughness,
       and overall usability of each of the Manual's sections:

                                                Neither
                                                Dissatisfied
                         Very                   nor                   Very       No
                         Dissatisfied   Dissatisfied Satisfied   Satisfied    Satisfied   Opinion
  Introduction

  Section 1.1: Requirement      Q          O         O         O         O         O
  for Renewable Fuels
  Section 1.2: Renewable        Q          O         O         O         O         O
  Fuel Standard Program
  Section 1.3: Brief
  Overview of Ethanol          O          O         O         O         O         O
  Production Process
  Chapter 1: What Laws Apply When I'm Constructing or Modifying a Plant
Section 1.1: National
Environmental Policy Act
Section 1.2: Clean Air Act
Section 1.3: Clean Water
Act
Section 1.4: Safe
Drinking Water Act
Section 1.5: Pollution
Prevention Act
Chapter 2: What Laws Apply
Section 2.1: Clean Air Act
Section 2.2: Emergency
Planning and Community
Right to Know Act
Section 2.3: Clean Water
Act
Section 2.4: Safe
Drinking Water Act
O
O
O

O
O

to Operating
O

O

O

O
O
O
O

O
O

an Ethanol
O

O

O

O
O
O
O

O
O

Plant?
O

O

O

O
O
O
O

O
O


O

O

O

O
O
O
O

O
O


O

O

O

O
O
O
O

O
O


O

O

O

O
November 2009                                                                       Page 8

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                Report Appendices
                                                    Neither
                                                    Dissatisfied
                           Very                     nor                    Very        No
                           Dissatisfied   Dissatisfied Satisfied    Satisfied    Satisfied    Opinion
   Section 2.5: Resource
   Conservation and
   Recovery Act -
   Hazardous Waste
   Appendices
   Appendix A (First Half):
   Summary of Laws Pertaining
   to Ethanol Production and
   Contacts
   Appendix A (Second Half):
   Contact Information
   Appendix B: Emergency
   Planning
   Appendix C: Clean Air Act
   Section 112-R Requirements
   Appendix D: Definitions Of
   Acronyms
   Appendix E: The NEPA Process
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
Do  you have any specific suggestions for improving the usefulness of any of the Manual sections?
(optional)
If you have not done so already, please remember to fill  out the final section of the main part of the
survey starting with "Part 4: Your Overall Satisfaction with the Manual" on page 3. Thank you!

Burden Statement: Public reporting burden for this collection of information is estimated to average fifteen (15)
minutes per response, including the time for reviewing instructions,  gathering information, and completing and
reviewing the collection of information, with a possible ten (10) additional minutes for the optional section at the
end. Send comments on the Agency's need for this information, the  accuracy of the provided burden estimates,
and any suggestions for reducing the burden, including the use of automated collection techniques to the Director,
OEI Collection Strategies Division, United States Environmental  Protection Agency (Mail  Code  2822T),  1200
Pennsylvania Avenue, Washington, DC 20460; and to the Office of Information and  Regulatory Affairs, Office of
Management and Budget, 725 17th Street NW, Washington, DC 20503, Attention: Desk Officer for EPA. Include the
EPA  ICR   number  (1711.04)   and  the  OMB  control   number  (2090-0019)   in  any  correspondence.
November 2009
                                                             Page 9

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                Report Appendices
Appendix B.   Customer Satisfaction Survey Results

Note: Questions that asked for optional contact information are not listed here, nor are any participant
comments that indicated which surveys responded to the survey.
       PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER
   1.  Prior to receiving this questionnaire, were you aware that EPA developed the Compliance
       Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available?

                                                       Response    Response
                                                        Percent       Count
        Yes                                              72.7%          16
        No                                              27.3%          6
                                               answered question           22
                                                 skipped question            2

   2.  Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities,
       published by EPA in  November 2007?  If No, please skip to Question 17.

                                                       Response    Response
                                                        Percent       Count
        Yes                                             100.0%         22
        No                                              0.0%           0
                                               answered guest/on           22
                                                 skipped question            2

   3.  Please check the method below that indicates how you learned about the Manual
                                                       Response    Response
                                                        Percent       Count
        Postcard distributed from EPA                        13.0%          3
        EPA  news release                                   4.3%           1
        EPA  radio or TV interview                            0.0%           0
        EPA  Web site                                      21.7%          5
        State contact                                      17.4%          4
        Meetings / conferences (e.g., FEW workshop,
        CenSARA's National Environmental Biofuels             13.0%          3
        Conference)
        Other correspondence from EPA                      8.7%           2
        Other method (please specify)                        17.4%          4
        Don't know/don't remember                         4.3%           1
        Other (please specify)
            •  From corporate office
            •  From this  survey and this consulting firm                       4
            •  This questionnaire
            •  From Ross & Associates
November 2009                                                                   Page 10

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
              Report Appendices
                                                  answered question
                                                    skipped question
                     23
                      1
    4.  Why were you interested in receiving the Manual? (check all that apply)

                                                          Response
                                                           Percent
         Wanted to learn more about applicable Federal laws
         for construction of facilities
         Wanted to learn about applicable Federal laws for
         operation of facilities
         General reference
         Other (please specify)
         I received the Manual without asking for it
                                                   answered question
                                                    skipped question
22.7%

63.6%

54.5%
4.5%
            Response
              Count
14

12
 1
 3
     22
      2
       PART 2: ACCESSING THE MANUAL
    5.  Please check the method below that indicates how you accessed the Manual.
         Downloaded the Manual from the EPA Web site
         Received a Compact Disc (CD) from EPA
         Received a CD from a state contact
         Received a CD from another contact outside of my
         facility
         Received a hard copy from EPA
         Received a hard copy from a state contact
         Received a hard copy from a different (not
         state/EPA) contact outside of my facility
         Other method (please specify:)
             •   Received electronic copy from corporate office
             •   Electronic copy from state contact
             •   Received link from Ross & Associates
             •   Emailed to me by EPA contact
                                                  answered question
                                                    skipped question
Response
Percent
73.7%
0.0%
0.0%
Response
Count
14
0
0
 0.0%

15.8%
 5.3%

 5.3%
3
1
                     19
                      5
    6.  Please rate how satisfied you were with the ease of accessing the Manual.
         Very Dissatisfied
         Dissatisfied
         Neither Dissatisfied nor Satisfied
         Satisfied
         Very Satisfied
Response
Percent
0.0%
0.0%
4.3%
43.5%
39.1%
Response
Count
0
0
1
10
9
November 2009
                         Page 11

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                                                 Report Appendices
         No Opinion
                                        ^^      13.0%
                                         answered question
                                           skipped question
                                                                                23
                                                                                 1
       Please describe any ways you think EPA could improve access to the Manual (optional):
       No Answers

    1.  Have you read any part of the Manual? If No, please skip to Question 17. If Yes, please continue
       to Part 3 of the survey.

                                                          Response     Response
                                                           Percent       Count
                                                            91.3%           21
Yes
No
                                                 ^^      8.7%
                                                  answered question
                                                    skipped question
                                                                       23
                                                                         1
       PART 3: STRUCTURE AND READABILITY OF THE MANUAL
    8.  Please rate your satisfaction with the Manual's readability.
         Very difficult to read
         Difficult to read
         Neither easy nor difficult to read
         Somewhat easy to read
         Very easy to read
         No opinion
    9.  Please rate the ease of navigating through the Manual to get to areas of interest.
         Very difficult to navigate to areas of interest
         Difficult to navigate to areas of interest
         Not easy, but not difficult to navigate to areas of
         interest
         Easy to navigate to areas of interest
         Very easy to navigate to areas of interest
         No opinion
Response
Percent
0.0%
0.0%
4.8%
47.6%
47.6%
0.0%
answered question
skipped question
Response
Count
0
0
1
10
10
0
21
3
Response
Percent
0.0%
0.0%
Response
Count
0
0
                                                   23.8%
                                                   57.1%
                                                   19.0%
                                                   0.0%
                                         answered question
                                           skipped question
12
4
0
                                                                                21
                                                                                 3
November 2009
                                                                           Page 12

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
   10. How would characterize the amount of information presented in the Manual?
         Significantly too little
         Slightly too little
         Right Amount
         Slightly too much
         Significantly too much
         No opinion
Response
Percent
0.0%
38.1%
57.1%
4.8%
0.0%
0.0%
answered question
skipped question
Response
Count
0
8
12
1
0
0
21
3
       We welcome any comments on your response:

       No Answers
I IV. PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL 1
11. a. Please rate your satisfaction



How well the Manual
met your needs for
understanding
compliance information
How clear the
information provided in
the Manual was on
where to go for more
information
The information
thoroughness of the
Manual

Very
Dissatisfied


0 (0%)



0 (0%)


0 (0%)
with the following:
Neither
Dissatisfied Di^atisfied
Satisfied

0 (0%) 5 (23.8%)



0 (0%) 6 (28.6%)


1 (4.8%) 5 (23.8%)


Satisfied


13
(61.9%)


1 1
j. j.
(52.4%)


13
(61.9%)


Very
Satisfied


3
(14.3%)


4
I
(19.0%)


2 (9.5%)


No
Opinion


0 (0%)



0 (0%)


0 (0%)
                                                               answered quest/on       21
                                                                 skipped question        3
       We welcome any comments on your response
          •   Should have provided some more details.
November 2009
          Page 13

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
  Report Appendices
       Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If
       so, we welcome any explanation and suggestions for improving your satisfaction with that
       section or chapter below.
                                                           Response
                                                            Percent
         Yes                                                 9.5%
         No                                                  90.5%
                                                   answered question
                                                     skipped question
Response
  Count
    2
    19
        21
          3
       We welcome any comments on your response:
           •  The section on air quality regulations needs improvement and updating.
           •  Construction to operations transition

       What else should EPA consider to improve your satisfaction with the Manual as a compliance
       assistance tool?
           •  A section targeted to the regulators (EPA/states) where the ethanol plants can provide
               input on details of plant operations and the importance of operational flexibility.
           •  Good overviews. Wish it was more thorough or provided examples. Is there a list of
               resources or contractors/consultants that could be provided for each state and what
               services they provide?
           •  I really can't think of anything, this was a very comprehensive manual.
           •  Include checklists, one-page "at-a-glance" tables, etc.
           •  The more specific and less vague it is, the more valuable a resource it will be.
           •  No suggestions at this time.
           •  Nothing
    12. Please rate your overall satisfaction with the Manual as a compliance assistance tool.
         Answer Options
         Very Dissatisfied
         Dissatisfied
         Neither Dissatisfied nor Satisfied
         Satisfied
         Very Satisfied
         No Opinion
Response
Percent
0.0%
0.0%
14.3%
76.2%
9.5%
0.0%
answered question
skipped question
Response
Count
0
0
3
16
2
0
21
3
November 2009
            Page 14

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
       PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE
    13. If EPA were to update the Manual, which update frequency would be the most useful?
         Answer Options
         Annually
         Every 3 years
         Every five years
         Other timeframe (please specify)
         I don't intend to  refer to the Manual again
         Other (please specify)
             •   When regulations change
             •   Any time there has been a major regulatory change
             •   As often  as changes are made
                                                  answered question
                                                    skipped question
Response
Percent
28.6%
47.6%
9.5%
14.3%
0.0%
Response
Count
6
10
2
3
0
      21
       3
    14. What other tools and approaches would you prefer for sharing compliance assistance
       information concerning ethanol facilities?
           •   Yearly EPA meetings with industry - compliance/update seminars
           •   Think it has been good so far.
           •   Workshops/meetings with open, unbiased, and frank discussions between regulators
              and ethanol plants.
           •   Email listserv for ethanol industry contacts that sends regulatory/policy changes
              applicable to the industry or NAICS code. Send industry compliance advisories that give
              helpful info on what are some compliance concerns along with any compliance
              assistance info or best practices.
           •   An associated question and answer document to is mainted so you could access it to see
              if there are  any changes to the manual.
           •   EPA inspection without fines (voluntary)
           •   Email updates/newsletters
           •   An email newsletter with  reminders, updates and new information would be helpful.
           •   new regulations and laws that come up every year
           •   No suggestions.  Website and manual are both sufficient in my opinion.
VI.    FINAL QUESTIONS!
    15. What category best describes your position?
         Answer Options
         Facility manager
         Facility compliance officer
         Other facility role (please specify)
         Contractor
Response
Percent
19.0%
61.9%
14.3%
4.8%
Response
Count
4
13
3
1
November 2009
          Page 15

-------
 Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                      Report Appendices
         If other:
             •   EH&S Manager (2)
             •   Consultant
             •   Environmental engineer/EHS manager
             •   Environmental, Health and Safety Coordinator
                                                 answered question
                                                  skipped question
                                           21
                                             3
    16.  Can we identify your facility or company (by name) as having participated in this survey?

         Answer Options
         Yes
         No
                      Response
                       Percent
                        52.2%
              ^^     47.8%
               answered question
                skipped question
                                 Response
                                   Count
                                     12
                                     11
                                                                             23
                                                                               1
 Optional Section: if you would like to provide more specific feedback on each Section of the

 Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer

 the following questions.


    1.  Please rank your overall satisfaction with the ease of understanding, information thoroughness,
        and overall usability of each of the Manual's sections:
Answer Options


Section I.I:
Requirement for
Renewable Fuels
Section 1.2:
Renewable Fuel
Standard Program
Section 1.3: Brief
Overview of Ethanol
Production Process
Answer Options

Section 1.1: National
Environmental Policy
Act
Section 1.2: Clean
Dissatisfied


  0(0%)

  0 (0%)
               Neither

Dissatisfied  «-£"-  Satisned   ^sfTed  Option
                           (28.6%)   1 (14-3%)   (42.9%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
2 (28.6%)
Satisfied
1 (14.3%)
1 (14.3%)
2 (28.6%)
                                      °(0%)
                                                (28.6%)
                                                 Neither
Dissatisfied   «-??«   Satisfied  JSL  Op^on
                                   0 (0%)

                                   questic
                           skipped question


                                    Very
                                                (28.6%)

                            answered question        7
                                                     17
0(0%)

0 (0%)
               Satisfied

                0(0%)

               1 (14.3%)
(71.4%)    °(0%)    (28.6%)
 November 2009
                                               Page 16

-------
 Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Air Act
Section 1.3: Clean
Water Act
Section 1.4: Safe
Drinking Water Act
Section 1.5:
Pollution Prevention
Act
Answer Options

Section 2.1: Clean Air
Act
Section 2.2:
Emergency Planning
and Community Right
to Know Act
Section 2.3: Clean
Water Act
Section 2.4: Safe
Drinking Water Act
Section 2.5: Resource
Conservation and
Recovery Act -
Hazardous Waste
Answer Options

Appendix A (First
half): Summary of
Laws Pertaining to
Ethanol  Production
and Contacts
Appendix A (Second
half): Contact
Information
Appendix B:
Emergency Planning
Appendix C: Clean Air
Act Section 112-r
requirements
Appendix D:
Definitions of
0 (0%)
0 (0%)
0 (0%)

Very
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)

Very
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)

Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)

Dissatisfied
1 (14.3%)
0 (0%)
1 (14.3%)
0 (0%)
0 (0%)
1 (14.3%)
3 (42.9%)
1 (14.3%)

Neither
Dissatisfied
nor
Satisfied
1 (14.3%)
1 (14.3%)
1 (14.3%)
1 (14.3%)
1 (14.3%)

Neither
Dissatisfied
nor
Satisfied
1 (14.3%)
2 (25.0%)
2 (28.6%)
3 (42.9%)
1 (14.3%)
(42.9%) (14.3%) (28.6%)
222
(28.6%) (28.6%) (28.6%)
(28.6%) ° (0%) (28.6%)
4 2
(57.1%) °(0/o) (28.6%)
answered question 7
skipped question 17
Satisfied Satisfied Opinion
6
fQC -JO/ \ 0(0 /o) 0(0/0)
(O3./ /O)
5
(85.7%)
(71.4%) (14.3%) 0(°°/o)
(71.4%) ° (0%) (14.3%)
(71.4%) (14.3%) °(°0/0)
answered question 7
skipped question 17
-_,.. f. . Very No
satisfied Satisfied opinion
2(28.6%) (4230/o) 0(0%)
2(25.0%) (5040%) 0(0%)
4 (57.1%) 0 (0%) 0 (0%)
4 (57.1%) 0 (0%) 0 (0%)
5(71.4%) M * 0(0%)
 November 2009
          Page 17

-------
 Evaluation of the Effectiveness of an Ethanol Compliance Manual
                                              Report Appendices
Acronyms
Appendix E: The NEPA
Process
0(0%)       0(0%)      4(57.1%)   3(42.9%)    0(0%)    0(0%)
                                      answered question       8
                                       skipped question      16
 Do you have any specific suggestions for improving the usefulness  of any of the Manual sections?
 (optional)
         Answer Options

         Yes
         No
         If yes, please provide:
                               Response
                                Percent
                                  0.0%
                                 100.0%

                        answered question
                          skipped question
Response
  Count
    0
    8
    0
         8
        16
 November 2009
                                                        Page 18

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices
Appendix C.   Interview Guide for Ethanol Facilities
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview.  The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans.  Rather, the evaluation is designed to
collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to
ethanol facilities.  For the interview, it is  not expected that you will have carefully read the entire Manual,
but please at a minimum review those parts of the manual that pertain to your area of work. The manual
can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol  plants  manual.pdf

If you have an questions, please contact Anna Williams or Amy  Wheeless, Ross & Associates  by phone
at 206-447-1805 or by email at anna.williams@ross-assoc.com or amy.wheeless@ross-assoc.com.
TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL


Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?

    1.  Where did you obtain the Manual?

    2.  How could distribution of or access to the Manual be improved?

TOPIC AREA II:  EFFECTIVENESS OF THE MANUAL


Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?

    1.  How readable and comprehendible is the Manual?

    2.  How comprehensive is the Manual in its scope?

    3.  What was  most/least useful or informative to you in the Manual?

    4.  Is the  contact information provided in the Appendices helpful? How could it be improved?

    5.  How has your facility used the Manual? If your facility has not used the Manual, why not?

    6.  What regulatory actions did your facility take, or intend to take,  as a result of reviewing the
       Manual?
November 2009                                                                      Page 19

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices
    7.  What process or management changes did your facility make or intend to make as a result of
       reviewing the Manual?

    8.  Did your facility make any changes in constructing or establishing the plant as a result of
       reviewing the Manual?

TOPIC AREA III: UNDERSTANDING AND  BEHAVIOR CHANGE

Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?

    1.  After reviewing the Manual, do you feel more aware of the applicable Federal regulations?

    2.  Are there still federal laws or regulations that you do not understand after reading the Manual?

    3.  Have there been any changes in your compliance results since reviewing the Manual?

    4.  What "beyond compliance" measures has your facility made as a result of using the manual?

    5.  Are there specific environmental health outcomes (e.g. amount of waste generated) that have
       resulted from use of the Manual?

    6.  Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that
       have resulted from this Manual,  or are you aware of any potential effects on human health?

TOPIC AREA IV: IMPROVING THE MANUAL

Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?

    1.  What changes  would you recommend to the Manual to improve its readability and navigation?

    2.  What could be  changed about the Manual to make it more thorough and comprehensive and to
       make it more effective as a compliance tool?

    3.  Are there topics you would like more or new information about in a future Manual?

    4.  What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
       or make facilities and other stakeholders aware that an updated Manual is available?

    5.  Do you have any other suggestions for improving the Manual?

TOPIC AREA V: OTHER COMPLIANCE  ASSISTANCE

Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?
November 2009                                                                        Page 20

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices
    1.  What are other ways you find information about regulations and applicable laws?

    2.  What are other ways EPA could present the information in the Manual to its targeted audiences?
       (e.g. workshops, interactive Web site)

FINAL QUESTION

    1.  Do you have any final "closing thoughts" or recommendations regarding the overarching
       questions, sub questions or the compliance assistance Manual in general?

                                 Thank you for participating!
November 2009                                                                      Page 21

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
Appendix D.   Interview Guide for Ethanol Contractors
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans.  Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol  plants manual.pdf

If you have an questions, please  contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.

TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL

Overarching Question:  How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?

    1.   Where did you obtain the Manual?

    2.   How could distribution of or access to the Manual be improved?

TOPIC AREA II: EFFECTIVENESS OF THE MANUAL

Overarching Question:  How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?

    1.   How readable and comprehendible is the  Manual?

    2.   How comprehensive is the Manual in its scope?

    3.   What was most/least useful or informative to you in the Manual?

    4.   Is the contact information provided in the Appendices helpful? How could it be improved?

TOPIC AREA III: UNDERSTANDING AND BEHAVIOR  CHANGE

Overarching Question:  Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?

    1.   After reviewing the Manual, do you feel more aware of the applicable Federal regulations?
November 2009                                                                         Page 22

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
    2.  Are there still federal laws or regulations that you do not understand after reading the Manual?

    3.  Are there specific environmental health outcomes (e.g. amount of waste generated) that have
       resulted from use of the Manual?

    4.  Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that
       have resulted from this Manual, or are you aware of any potential effects on human health?

TOPIC AREA IV: IMPROVING THE MANUAL

Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?

    1.  What changes would you recommend to the Manual to improve its readability and navigation?

    2.  What could be changed about the Manual to make it more thorough and comprehensive and to
       make it more effective as a compliance tool?

    3.  Are there topics you would like more or new information about in a future Manual?

    4.  What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
       or make facilities and other stakeholders aware that an updated Manual is available?

    5.  Do you have any other suggestions for improving the Manual?

TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE

Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?

    1.  What are other ways you find information about regulations and applicable laws?

    2.  What are other ways EPA could present the information in the Manual to its targeted audiences?
       (e.g. workshops, interactive Web site)

FINAL QUESTION

    1.  Do you have any final "closing thoughts" or recommendations regarding the overarching
       questions, sub questions or the compliance assistance Manual in general?

                                 Thank you for participating!
November 2009                                                                       Page 23

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
Appendix E.    Interview Guide for State Compliance Staff
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans.  Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol  plants manual.pdf

If you have any questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.

TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL

Overarching Question:  How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?

    1.   Where did you obtain the Manual?

    2.   Does your agency refer ethanol facilities and contractors to the Manual?

    3.   How could distribution of or access to the Manual be improved?

TOPIC AREA II: EFFECTIVENESS OF THE MANUAL

Overarching Question:  How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?

    1.   How readable and comprehendible is the  Manual?

    2.   How comprehensive is the Manual in its scope?

    3.   Is the contact information provided in the Appendices helpful? How could it be improved?

    4.   Have you been contacted by an ethanol facility or contractor as a result of the Manual?

TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE

Overarching Question:  Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?



November 2009                                                                         Page 24

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                 Report Appendices



    1.  Could compliance with state laws and rules be enhanced using the Manual? How?

    2.  In your interactions, do facilities seem aware of the applicable Federal regulations?

    3.  Have you noticed any changes in facility compliance with regulations since the Manual was
       released?

TOPIC AREA IV: IMPROVING THE MANUAL

Overarching Question: How could the Manual be  improved to provide effective compliance assistance to ethanol
facilities?

    1.  What changes would you recommend to the Manual to improve its readability and navigation?

    2.  What could be changed about the Manual to make it more thorough and comprehensive and to
       make it more effective as a compliance tool?

    3.  Are there topics you would like more or new information about in a future Manual?

    4.  What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
       or make facilities and other stakeholders aware that an updated Manual is available?

    5.  Do you have any other suggestions for improving the Manual?

TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE

Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?

    1.  What are other ways EPA could present the information in the Manual to its targeted audiences?
       (e.g. workshops, interactive Web site)

FINAL QUESTION

    1.  Do you  have any final  "closing  thoughts" or  recommendations  regarding  the  overarching
       questions, sub questions or the compliance assistance Manual in general?

                                 Thank you for participating!
November 2009                                                                        Page 25

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
Appendix F.    Interview Guide for EPA Staff
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans.  Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will  have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work.  The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf

If you have any questions, please contact Anna Williams or Amy Wheeless,  Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.

TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL

Overarching Question:  How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?

    1.   Does your agency refer ethanol facilities and contractors to the Manual?

    2.   How could distribution of or access to the Manual be improved?

TOPIC AREA II: EFFECTIVENESS OF THE MANUAL

Overarching Question:  How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?

    1.   How readable and comprehendible is the Manual?

    2.   How comprehensive is the Manual in its scope?

    3.   Is the contact information provided in the Appendices helpful? How could it be improved?

    4.   Have you been contacted by an ethanol facility or contractor as a result of the Manual?

TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE

Overarching Question:  Has the existence and use of the Manual led to  increased understanding and behavior
change to support improved compliance?

    1.   In your interactions, do facilities seem aware of the applicable  Federal regulations?
November 2009                                                                          Page 26

-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual                  Report Appendices
    2.  Have you noticed any changes in facility compliance with regulations since the Manual was
       released?

TOPIC AREA IV: IMPROVING THE MANUAL

Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?

    1.  What changes would you recommend to the Manual to improve its readability  and navigation?

    2.  What could be changed about the Manual to make it more thorough and comprehensive and to
       make it more effective as a compliance tool?

    3.  What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
       or make facilities and other stakeholders aware that an updated Manual is available?

    4.  Do you have any other suggestions for improving the Manual?

TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE

Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?

    1.  What are other ways EPA could present the information in the Manual to its targeted audiences?
       (e.g. workshops, interactive Web site)
FINAL QUESTION

    1.  Do you  have any  final  "closing  thoughts" or  recommendations  regarding the overarching
       questions, sub questions or the compliance assistance Manual in general?

                                 Thank you for participating!
November 2009                                                                       Page 27

-------