United States Office of Policy,
Environmental Protection Economics and Innovation
Agency (1807T)
November 2009
EPA-100-R-10-001
Evaluation of the
Effectiveness of an
Ethanol Compliance
Manual
Final Report
Promoting Environmental Results
4
Through Evaluation
-------
Acknowledgements
This evaluation was performed by Ross & Associates Environmental Consulting, Ltd. (Ross
& Associates) under contract to Industrial Economics, Incorporated (lEc) for EPA's Office of
Policy, Economics and Innovation (OPEI) under Contract EP-W-07-028 between EPA and
I EC. The evaluation team included Anna Williams, Tamara Power-Drutis and Amy Wheeless
of Ross & Associates, Ashley Betts and Ward Burns of EPA Region 7 and Michelle
Mandolia of OPEI's Evaluation Support Division.
This report was developed under the Program Evaluation Competition, sponsored by OPEI.
To access copies of this or other EPA program evaluations, please go to EPA's Evaluation
Support Division's website at http:www.epa.gov/evaluate.
-------
Table of Contents
Executive Summary i
I. Introduction 1
Background 1
Purpose of the Evaluation 2
Report Organization 2
II. Methodology 3
Evaluation Approach 3
Information Collection and Analysis 4
III. Findings 8
Topic Area I: Outreach and Access 8
Topic Area II: Effectiveness of the Manual as an Information Tool 11
Topic Area III: Changes in Understanding and Behavior 14
Topic Area IV: Improving the Manual 16
Topic Area V: Other Compliance Assistance 19
IV. Recommendations 20
Appendices (Available in Separate Document)
A. Customer Satisfaction Survey
B. Customer Satisfaction Survey Results
C. Interview Guide - Ethanol Facility
D. Interview Guide-Contractor
E. Interview Guide - State Compliance Staff
F. Interview Guide - EPA Staff
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Executive Summary
Background and Evaluation Process
Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were
issued new environmental permits. The increase in new ethanol facilities was especially significant given
that, during the previous 20 years, only 30 ethanol plants had been permitted. Region 7 recognized that
the ethanol production sector may need targeted outreach and education on the pertinent federal
environmental rules, regulations, and programs that govern the sector's construction and operation.
To meet this need, in November 2007, Region 7 staff published a compliance assistance manual for new
ethanol facilities entitled Environmental Laws Applicable to Construction and Operation of Ethanol Plants
(hereto after referred to as "the Manual"). The Manual included information on federal environmental
programs and the roles of state and federal authorities in overseeing ethanol facility construction and
operation. The Manual was primarily targeted to ethanol plants, but was also likely of interest to state
regulatory program offices, trade associations, consultants, and the public. Region 7 conducted a
number of outreach activities to publicize the Manual to ethanol plants and other stakeholders.
Region 7, with support from the EPA Office of Policy, Economics and Innovation (OPEI), wished to
evaluate the success of the Manual in improving industry compliance with relevant rules and regulations
and to gather information on the Manual's readability, quality of information, and overall usefulness as a
compliance assistance tool to ethanol facilities in the Region. In addition, Region 7 wanted to identify
ways to improve facility satisfaction with the Manual as a compliance assistance tool and to determine
what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of
this evaluation are intended primarily for Region 7 staff as they prepare revisions to the Manual and
produce other outreach materials for the ethanol production sector.
An EPA-contractor team led the evaluation. Two representatives from Region 7 were part of the team;
one who had helped to develop the Manual as a member of the Region 7 Biofuels Team, and one
representative of the Region's policy and management division. The other EPA team member was an
evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI. The contractor team
Exhibit ES-i. Overarching Evaluation Questions
Topic Area I: Outreach and Access. How effective are Region 7's efforts to advertise the
availability of and distribute the Manual to ethanol facilities and other intended audiences?
Topic Area II: Effectiveness of the Manual as an Information Tool. How effective is the
Manual as a tool for transferring compliance (and related facility technical support) information to
ethanol facilities?
Topic Area III: Changes in Understanding and Behavior. Has the existence and use of the
Manual led to increased understanding and behavior change to support improved compliance?
Topic Area IV: Improving the Manual. How could the Manual be improved to provide effective
compliance assistance to ethanol facilities?
Topic Area V: Other Compliance Assistance. What approaches other than the Manual would
provide effective compliance assistance to the ethanol facilities?
November 2009 ES-i
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
consisted of Industrial Economics (lEc) and a subcontractor, Ross & Associates Environmental
Consulting, Ltd. (Ross & Associates). Ross & Associates implemented most of the contractor
responsibilities and produced this report with support from the other team members.
The evaluation focused on five overarching evaluation questions, listed on the previous page in Exhibit
ES-1. To answer the overarching questions, the evaluators conducted 20 interviews with facility
environmental managers, ethanol industry contractors, state compliance staff members, and EPA staff
members. In addition, the evaluators developed a customer satisfaction survey and distributed the
survey to ethanol facilities in Region 7 and ethanol industry contractors known to work in the Region.
Finally, the evaluators conducted research on ethanol plant compliance to try to answer the overarching
evaluation question on the effect the Manual had on facility compliance.
Findings
Those interviewed and surveyed for the evaluation found the Manual to be very useful. They appreciated
its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the proactive
assistance approach embodied in the Manual represented. Generally, comments for improvement
focused on timelier and improved distribution and a variety of changes to make the Manual a stronger
compliance assistance tool. Below were the major findings for each overarching evaluation topic area.
Topic Area I: Outreach and Access
Awareness of the Manual amongst the primary audiencesethanol facilities and contractorswas not as
prevalent as it could be despite outreach efforts by Region 7 staff. In addition, the timing of the Manual's
publication occurred after most new plants had commenced or completed construction; only a handful of
new ethanol plants started productions since November 2007. A majority of the facilities that the
evaluators spoke with in the summer of 2009 were not immediately aware of the Manual's existence.
Nonetheless, most of these facilities were receptive to reviewing the Manual as a part of this evaluation.
Feedback also indicated that the audience for the Manual is larger than just Region 7-specific ethanol
facilities and contractors, and that promoting the Manual nationally could be beneficial, particularly to EPA
Regions with significant numbers of ethanol plants.
Topic Area II: Effectiveness of the Manual as an Information Tool
Participants in the evaluation found the Manual to be well organized, easy to navigate, and an appropriate
tool for conveying compliance information to facilities. Having this material, including appropriate
contacts, organized in one place provided for an easy reference for facility managers and they found the
Manual to be readable and organized logically. Participants provided varied and inconsistent feedback
on whether the Manual was too technical or not technical enough, but everyone interviewed and surveyed
indicated that they found the Manual to be useful in some way. Whether the Manual has the appropriate
level of detail likely depends on the reader's existing level of knowledge and facility needs.
Topic Area III: Changes in Understanding and Behavior
All of the facilities and contractors that participated in this evaluation said the Manual was comprehensive
and was helpful for increasing or solidifying their understanding of environmental compliance
requirements.
This evaluation cannot conclusively say that the Manual has led to any significant changes that would
affect facility compliance. With only a few exceptions, respondents to this evaluation were not able to
identify specific instances where facilities had more awareness of applicable federal regulations or had
made compliance or management changes as a result of the Manual. Research of the EPA Enforcement
November 2009 ES-ii
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
and Compliance History Online (ECHO) database showed that, after the Manual was published in late
2007, there was some change in informal and formal enforcement actions at Region 7 ethanol facilities,
but due to lagging and incomplete enforcement data and the inability to know exactly when a facility
reviewed the Manual, the evaluators cannot say conclusively than the Manual had an effect on
compliance.
Topic Area IV: Improving the Manual
Respondents to this evaluation had a number of suggestions on how future versions of the Manual could
be improved. These suggestions centered around additional content, such as a section on cellulosic
ethanol, or around functional content, such as checklists, case studies, and specific examples. The
recommendations below carry forward many of the suggestions offered by those interviewed and
surveyed.
Topic Area V: Other Compliance Assistance
Evaluation participants thought that, while the Manual itself was a useful tool, other venues, such as
conferences, EPA workshops, a more interactive Web site, and one-on-one interaction with ethanol
facilities were also valuable approaches for providing compliance assistance to ethanol facilities.
Recommendations
Evaluation participants clearly conveyed that the Manual was useful and valuable and that it effectively
communicated relevant information for ethanol plants. The following recommendations focus on ways to
improve the Manual for any potential future versions and on other ways that Region 7 could make the
existing information available to its intended audiences. The evaluators understand that there may be
limited resources available for updating the Manual and limited time available for staff to provide other
kinds of compliance assistance to ethanol facilities. Further, the market for ethanol has been shifting
dramatically and demand for new facility permits has not returned to the peak it was in when the Manual
was first released. Demand for the Manual and feasibility of implementing any of the evaluation's
recommendations will likely be influenced by these and other factors. The following recommendations,
which are expanded upon in the body of the report, are offered for Region 7's consideration.
1. As a first step, consider whether to tailor the Manual's content and distribution to the national level.
2. Consider additional outreach strategies and targeted audiences.
3. Consider making this tool primarily Web-based, with a hard copy Manual as a supplement.
4. Update the Manual to include new regulatory and compliance information.
5. Create an executive summary to encapsulate the main points of the Manual.
6. Incorporate more "at a glance" information, such as lists of resources, call-out boxes, checklists, and
calendars.
7. Provide more specific examples and case studies.
8. Expand content to include regulatory information about topics of interest to ethanol plants, such as
cellulosic ethanol production and byproduct diversion into other products.
9. Expand information on beyond compliance efforts and best practices.
10. Apply the lessons from this evaluation to the Region's other compliance assistance and beyond
compliance efforts.
November 2009 ES-iii
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
I. Introduction
Background
Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were
issued new environmental permits. The increase in new ethanol facilities was especially significant given
that, during the previous 20 years, only 30 ethanol plants had been permitted. Region 7 recognized that
the ethanol production sector may need targeted outreach and education to understand the pertinent
federal environmental rules, regulations, and programs that govern the sector's construction and
operation.
To meet this need, in November 2007, Region 7 staff published a
compliance assistance manual for new ethanol facilities entitled "The Manual in general was
Environmental Laws Applicable to Construction and Operation of a huge undertaking and a
Ethanol Plants (hereto after referred to as "the Manual"). The Manual great idea... a good public
included information on federal environmental programs and the roles | service..." - EPA contact
of state and federal authorities in overseeing ethanol facility
construction and operation.1
Ethanol plants were the primary audience for the Manual, but the Manual was also likely of interest to
state regulatory program offices, trade associations, consultants and the public. To distribute the Manual
to the relevant stakeholders, Region 7 staff conducted a number of outreach activities:
Staff announced the development of the Manual at the 2007 National Association of Farm
Broadcasting Convention and distributed ethanol materials to attendees and exhibitors.
Staff developed a news release about the development of the Manual and distributed it to those who
had expressed interest and to the editors of agriculture and biofuels magazines.
Staff posted the Manual on the EPA Region 7 Web site.
The Region 7 Office of Public Affairs (OPA) shared ethanol postcards with the Chairs of the Missouri
House Agriculture Committee, the Agribusiness Committees, and the Senate Agriculture Committee,
along with members of these committees, as well as at relevant conferences (e.g., Commodity Class,
the Missouri Agribusiness Association annual meeting, the Kansas Feed and Grain annual
conference, the Nebraska Governor's Conference on Agriculture, and the Missouri Governor's
Conference on Agriculture).
The Region 7 Regional Administrator (RA) and OPA attended the Fuel Ethanol Workshop and
announced the development of the Manual. OPA collected contact information for those interested in
the Manual and notified them via email when Region 7 released the Manual.
The RA conducted news interviews about the Manual (e.g., Missourinet, WHO-Farm radio) and
mentioned the Manual in a number of speeches (e.g., at Commodity Classic, Bunge North America,
Inc., CenSARA's National Environmental Biofuels Conference).
OPA met with executives of ethanol plants and shared information about the Manual.
1 A copy of the Manual can be downloaded at:
http://www.epa.qov/reqion07/priorities/aqriculture/ethanol plants manual.pdf
November 2009 Page 1
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Air program staff discussed the Manual at a number of events and meetings (e.g., Iowa State Field
Briefing, a meeting with the head of EPA's Office of Enforcement and Compliance Assistance,
CenSARA Director's Meeting).
Staff informed a number of interested stakeholders about the Manual, including RCRA (Resource
Conservation and Recovery Act) State programs, the U.S. Army Corps of Engineers. Staff also
informed two ethanol facilities about the Manual during the facilities' RCRA inspections.
Each Emergency Planning & Community Right-to-Know Act (EPCRA) state coordinator announced
the availability of the Manual at his or her State Emergency Response Commission meetings.
Purpose of the Evaluation
Region 7, with support from the EPA Office of Policy, Economics, and Innovation (OPEI), wished to
evaluate the success of the Manual in improving industry compliance with relevant rules and regulations
and to gather information on the Manual's readability, quality of information, and overall usefulness as a
compliance assistance tool to ethanol facilities in the Region. In addition, Region 7 wanted to identify
ways to improve facility satisfaction with the Manual as a compliance assistance tool and to determine
what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of
this evaluation are intended primarily for Region 7 staff as they prepare revisions to the Manual and
produce other outreach materials for the ethanol production sector.
An EPA-contractor team led the evaluation. Two representatives from Region 7 were part of the team;
one who had helped to develop the Manual as a member of the Region 7 Biofuels Team, and one
representative of the Region's policy and management division. The other EPA team member was an
evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI. The contractor team
consisted of Industrial Economics (lEc) and a subcontractor, Ross & Associates Environmental
Consulting, Ltd. (Ross & Associates). Ross & Associates implemented most of the contractor
responsibilities and produced this report with support from the other team members.
The results of this evaluation are intended primarily for Region 7 staff as they prepare revisions to the
Manual and produce other outreach material for the ethanol production sector. The results of this
evaluation could also assist this and other Regions with compliance assistance materials for industry and
manufacturing sectors.
Report Organization
This report consists of four sections, in addition to this Introduction (Section I). Section II provides an
overview of the evaluation methodology. Section III describes the findings for each evaluation question.
Section IV lists and describes the evaluation's concluding recommendations for Region 7. Appendices,
which include a copy of the survey, the survey results, and the interview guides, are provided in a
separate document.
November 2009 Page 2
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
II. Methodology
The following section provides an overview of the evaluation methodology, including the team's initial
approach for planning the evaluation and information collection and analysis process.
Evaluation Approach
As a first step in the evaluation, the evaluation team refined the initial evaluation questions that had been
submitted as a part of OPEI's evaluation competition, identified the information needs to answer the
refined questions, and updated the draft ethanol manual logic model that had been developed previously
to help inform the evaluation.
Evaluation Questions
The evaluation team developed the overarching evaluation questions listed below to structure the
evaluation. The overarching questions cover five topics: effectiveness of distribution of to the Manual,
effectiveness of the Manual as an information tool, effectiveness of the Manual on understanding and
behavior changes, possible improvements to the Manual, and other approaches for providing compliance
tools.
Exhibit i. Overarching Evaluation Questions
Topic Area I: Outreach and Access. How effective are Region 7's efforts to advertise the
availability of and distribute the Manual to ethanol facilities and other intended audiences?
Topic Area II: Effectiveness of the Manual as an Information Tool. How effective is the
Manual as a tool for transferring compliance (and related facility technical support) information
to ethanol facilities?
Topic Area III: Changes in Understanding and Behavior. Has the existence and use of
the Manual led to increased understanding and behavior change to support improved
compliance?
Topic Area IV: Improving the Manual. How could the Manual be improved to provide
effective compliance assistance to ethanol facilities?
Topic Area V: Other Compliance Assistance. What approaches other than the Manual
would provide effective compliance assistance to the ethanol facilities?
These overarching evaluation questions informed data collection, including development of the interview
guides and the survey (see Appendices). This report presents findings according to each overarching
evaluation question.
Evaluability Assessment
Following development of the evaluation questions, the evaluation team conducted an evaluability
assessment to ensure that there would be enough information to assess the effectiveness of the Manual.
This evaluability assessment identified:
What information would be needed to address the key evaluation questions;
Whether the needed information would be available; and
How the available information would be collected.
November 2009 Page 3
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
The conclusion of this evaluability assessment was that, with the available data and information collection
methods, to some extent all evaluation questions would be answerable, with qualitative, yet authoritative,
information and stakeholder feedback. The one exception was Topic Area III: the evaluability
assessment concluded that, due to a lack of data, the evaluation would not be able to draw definitive
conclusions about a causal relationship between the Manual and facility compliance.
Logic Model
In preparing for this evaluation, Region 7 developed a logic model for the Manual and its associated
activities. The evaluation team refined the model to elaborate on potential outcomes of Manual use (see
Exhibit 2 on the following page). The final logic model depicts the linkages between program activities
inputs, outputs, stakeholders, and expected outcomes of Region 7's outreach activities and compliance
assistance tool.
Information Collection and Analysis
In planning for the evaluation, the evaluation team determined that interviews with relevant stakeholders,
supplemented by a survey of facilities and contractors, would help to answer the overarching evaluation
questions. The primary target audience for the Manual and thus the main source of feedback were
ethanol facilities and contractors. The secondary audiences were state and EPA Region 7 compliance
staff. While trade associations and the public were also listed in the logic model as an "audience" or
"customers" for the Manual, they were not targeted for separate information collection in this evaluation.
Using information provided by Region 7, the evaluators contacted the ethanol plants and a handful of
contractors to assess interest in participating in an interview and/or a survey. During this contact, the
evaluators also offered to send Internet links for the Manual to those facilities and contractors who were
not previously aware of the Manual or had not received it.
In addition to interviews and surveys, the evaluators conducted some limited research on facility
compliance in the Region to better understand any changes in facility compliance.
Interview Approach
The evaluators collected data from ethanol plants, contractors, EPA staff, and state compliance staff
through a set of telephone interviews. Facility and contractor interviews were with individuals who
confirmed that they had received and reviewed the Manual. Region 7 staff contributed suggestions on
who to interview at EPA and state agencies. The list of interviewees was reviewed by other members of
the evaluation team to ensure a representative selection. The final set of interviewees consisted of:
Individuals at seven facilities that had responsibilities pertaining to environment compliance.
Three ethanol-industry environmental consultants/contractors; an additional consultant submitted
written feedback.
Five state compliance staff members from three Region 7 state environmental agencies.
Five EPA staff members, four of whom were from Region 7 and one from EPA Headquarters.
November 2009 Page 4
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
Exhibit 2. Ethanol Compliance Manual Logic Model
Goal: Improve environmental regulatory compliance among ethanol facilities in order to reduce impacts on environmental and human health
Resources from:
EPA Region 7
Biofuels Team
EPA Region 7
Ag Team
i State
Agencies
Trade Groups
ix
ACTIVITIES
Create
Compliance
Manual
Answer
Regulatory
Questions
Develop Other
Outreach
Materials
Improve and
Revise Manual
PARTICIPANTS
Ethanol
Facilities
Biofuels Team
and Regulatory
Programs
State
Compliance
Staff
Ethanol
Contractors
Trade Groups
i
ix
SHORT TERM
Improved Awareness and
Knowledge of Regulatory
Requirements and
Attitude Change
OUTCOMES
MEDIUM TERM
Changes in Ethanol
Facilities Operations or
Management
LONG TERM2
Improved Facility
Compliance
Reduced
Environmental and
Human Health
Impacts
OUTCOME MEASURES
# of facilities that say they
better understand how to
comply with regulations
# of facilities that
contacted someone for
further compliance
assistance
# of Manuals accessed
(print copies/ downloads)
# of facilities that took at
least one recommended
action to comply with
regulations
# of facilities that
adopted process
changes
# of facilities improving
environmental
management systems or
conducting reviews
Compliance rate
changes
#of facilities that
have changed
regulatory status
#of facilities that
eliminated, treated or
reduced emissions or
other pollutants
Quantified
environmental
improvements
It was assumed that increased compliance with environmental laws would result in the long-term outcomes of improved environmental and public health and that
these long-term outcomes would be realized over the course of months or years. It was beyond the scope or ability of this evaluation to identify whether these
outcomes have occurred as a direct result of the compliance assistance manual in question. Similarly, it was not feasible as part of this evaluation to gather the
data needed to develop the long-term (aspirational) performance measures listed in the logic model.
November 2009
Page 5
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Interviewees received interview guides (see Appendices C, D, E, and F) in advance of their interviews.
Interviews were generally conducted by one Ross & Associates staff member and lasted 15 to 30
minutes. The evaluators conducted the telephone interviews based on the interview guides; however, the
order in which topics were discussed sometimes varied. Notes from these interviews were collected
during the interviews and entered in an Excel workbook. These notes were considered confidential and
not included in this report. Any specific quotes that this report uses as examples were sent to the
relevant interviewees to ensure accurate communication of ideas, and are not attributed to the individual
in this report.
The evaluation team had originally considered conducting a set of follow-up interviews with the primary
and secondary audiences after doing an initial round of information collection. However, after conducting
the interviews, the evaluation team determined that follow-up interviews would be unlikely to yield
additional feedback or insight. However, the evaluation team did meet with members of the EPA Region
7 Biofuels Team to discuss the report's draft findings and recommendations.
Survey Approach
There were as many as 150 ethanol facilities in Region 7, as well as a number of ethanol industry
contractors, who may have received and read the Manual. To reach as many of them as possible and
understand their access to and satisfaction with the Manual, the evaluation team developed and
administered a written customer satisfaction survey to be completed by mail or on the Internet.
The Federal Paperwork Reduction Act of 1995 requires that federal agencies receive approval from the
Office of Management and Budget (OMB) before requesting information from more than nine non-federal
entities, and must do so by submitting an Information Collection Request (ICR) to OMB for approval. The
evaluation team developed a survey, submitted the survey under generic ICR 1711.05 to OMB, and
received approval following some minor survey revisions.
Following OMB approval of the survey, Ross & Associates emailed or mailed the survey to all Region 7
ethanol facilities and contractors for which it had contact information. This survey (see Appendix A)
focused on customer satisfaction with the Manual. Survey responses were anonymous unless
respondents chose to identify themselves.
Participants were encouraged to respond to the survey within approximately four weeks. Approximately
three weeks after the survey was mailed, the evaluators contacted facilities and contractors with a
reminder email or reminder postcard and extended the due date to solicit more responses.
The evaluators mailed or emailed surveys to approximately 120 facilities and contractors. Of these,
approximately ten were returned as having no recipient. Of the remaining, 24 facilities and contractors
completed surveys; at least three had also participated in a telephone interview. Others may have
participated in telephone interviews, but did not identify themselves in the survey.
The response rate for the survey was approximately 20 percent; generally, response rates for customer
satisfaction surveys range from 10 percent to 30 percent, so this response rate was not unexpected.
Factors that may have contributed to this response rate include the fact that the evaluators did not
connect with all facilities and contractors to alert a specific contact that a survey would be forthcoming;
some facilities in Region 7 were no longer in operation; and there was no tangible and immediate
incentive offered for completing the survey.
Additional Information Review
In addition to the surveys and interviews, the evaluators reviewed background material and researched
compliance information through available data to help answer the overarching evaluation questions. The
evaluators reviewed the Manual itself, the outreach activities that Region 7 had done to promote
November 2009 Page 6
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
distribution of the Manual, and the plant list available at Ethanol Producer Magazine
(http://ethanolproducer.com/plant-list.isp'), which provides status of operation, when the facility started
production, and type of feedstock. In addition, the evaluators conducted Internet searches to see which
organizations or companies had posted links to the Manual from their Web sites.
Finally, the evaluators reviewed EPA's Enforcement and Compliance History Online (ECHO) database.
ECHO provides an overview of a facility's environmental record under the Clean Air Act (CAA) Stationary
Source Program, the Clean Water Act (CWA) National Pollutant Elimination Discharge System (NPDES),
and the Resource Conservation and Recovery Act (RCRA), and associated state and federal responses.
The evaluators queried each facility in Region 7 that had information in the database and tracked informal
and formal violations pre-publication of the Manual and post-publication of the Manual (the Manual was
published in November 2007). The evaluators also looked at the compliance history of specific facilities
that were interviewed or surveyed and compared the results for a richer analysis of compliance.
November 2009 Page 7
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Findings
The following pages describe the evaluation's key findings, organized by the overarching evaluation
questions. Those interviewed and surveyed for the evaluation found the Manual to be very useful. They
appreciated its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the
proactive assistance approach embodied in the Manual represented. Generally, comments for
improvement focused on timelier and improved distribution and a variety of changes to make the Manual
a stronger compliance assistance tool.
Topic Area I: Outreach and Access
How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol
facilities and other intended audiences?
Overarching Findings
As discussed in Section I, Region 7 staff conducted a number of outreach activities to publicize the
availability of the Manual to ethanol plants and other interested stakeholders. However, awareness of the
Manual was not as prevalent as it could be for these audiences, and the timing of its publication was not
coincident with the largest increase in construction of ethanol plants. Nonetheless, access to the Manual
was sufficient for stakeholders who were aware of it, and many had additional suggestions for how to
improve advertisement of the Manual. Feedback also indicated that the audience for the Manual was
larger than just Region 7-specific ethanol facilities and contractors, and that promoting the Manual
nationally could be beneficial, particularly to EPA regions with significant numbers of ethanol plants.
/Access to Manual: Overall, for facilities and contractors aware of the Manual, access to the Manual
was considered sufficient, with many receiving it from an EPA or state contact or at a workshop or
conference. However, a majority of facilities (31 of 51) that the evaluators spoke to in the initial
evaluation contact period were not immediately aware of the Manual's existence. Nonetheless, most
of these facilities were receptive to receiving a Web link to the Manual. Of these 51 facilities, eight
had started production in 2008 or 2009; only one of these newer facilities indicated awareness of the
Manual prior to the evaluation.
Timing: Many interviewees noted that the timing of publication of the Manual was not idealfew
ethanol facilities have been constructed since November 2007, after the Manual was published, and
many have gone out of business or halted production since that time.
Distribution Audiences: Many facilities indicated that they use contractors and consultants to find out
information on current or upcoming regulations and refer to
them with questions on compliance or permitting. Facilities also I "Ethanol is not unique to Region
look to other ethanol-related organizations that interact regularly 7 |f a regjon comes out with
with the ethanol sector for information. These groups were something that's a good
noted as venues for distribution if EPA were to develop future product, then the agency should
versions of the Manual. consider marketing it beyond
National Relevance: Several interviewees noted that the tnat region... It should be a
information in the Manual would be well-received in other EPA national manual. The federal
Regions with significant numbers of ethanol plants. In addition, regulations are all the same, so
research during this evaluation found that the Manual was tney should all use the same
linked on the state environmental agency Web sites of states manual."-EPA contact
not in Region 7, highlighting that non-Region 7 ethanol plants
November 2009 Page 8
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
may have found this information to be relevant.
Interview Findings
Of those that participated in the interviews, six of the seven facilities and two of the three contractors were
aware of the Manual prior to being contacted for the evaluation. These interviewees indicated that they
had received it from: a contact at EPA Region 7; a manager; training sessions and workshops; and an
environmental consultant.
Generally, the timing of the publication of the Manual was more of an issue than mode of distribution in
getting information out to relevant stakeholders. As few ethanol plants have been placed into operation
since the Manual was published, many of the facilities the evaluators spoke with were already aware of
the information, having been in operation for some period of time. Region 7 noted that more than 100
facilities have been issued environmental permits between June 2005 and June 2008. According to
Ethanol Producer Magazine, only 11 facilities have been put into operation after November 2007,
indicating that the largest increase in activity was prior to publication of the Manual.3 Additionally, a
number of plants have gone out of business or have halted production in recent months. According to
Ethanol Producer Magazine, only seven facilities in Region 7 were not operating during the summer and
autumn of 2009. However, during the evaluation, the evaluators encountered disconnected lines,
returned surveys, and comments from ethanol plant staff that indicated that the number of facilities that
had at least temporarily halted production was higher.
Interviewees from the state agencies all work in some capacity with ethanol facilities and noted that they
have not referred any facilities to the Manual, but indicated that they may in the future as the ethanol
industry rebounds and new facilities are constructed. Both EPA and state staff noted that the information
in the Manual was helpful to operators just beginning to plan construction of a facility, but those people
are difficult to identify unless they approach regulators before beginning plant construction.
For improving distribution of the Manual, contractors were highlighted as an important and oft-used
resource for ethanol facilities on environmental issues. Five of the seven ethanol plants that were
interviewed indicated that they receive assistance on environmental compliance and permitting issues
from contractors and that these contractors alert them of new and upcoming regulations. All of the
contractors also stated that they help many ethanol plants in Region 7 with their environmental
compliance issues.
In addition to contractors and consultants, facility interviewees noted that they received information about
environmental regulations and tools from many different sources, and suggested these venues as ways
to get information about to this audience in the future. Region 7 distributed the Manual using some of the
below approaches (e.g., making information about the Manual available at workshops, alerting relevant
stakeholders), but feedback suggested that wider distribution through a number of venues, including the
following, would be useful:
Make Manual available at environmental workshops.
Provide Manual during EPA or state on-site technical assistance or inspections.
Make information available on a more interactive EPA Web site.
Distribute information on EPA and external email listservs.
Post Manual on state environmental agency Web sites.
Send hard copies of Manual to all potentially relevant organizations or facilities, including:
3 Ethanol Producer Magazine, Plant List. Accessed at http://ethanolproducer.com/plant-list.isp on August 18, 2009.
November 2009 Page 9
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
o Directly to contractors and consultants.
o State regulatory agencies.
o State ethanol boards.
o EPA program offices.
o National and state ethanol trade associations.
o National and state environmental consulting trade associations.
o Air resource agencies.
Finally, two EPA staff members and two contractors noted, unprompted, that the information in the
Manual is not just relevant to Region 7 ethanol plants, and the Manual should be distributed and
advertised more nationally.
Survey Findings
Overall, survey respondents were satisfied with their access to the Manual and had generally heard about
it either directly from EPA, from a state contact, or during a workshop or conference.
Four respondents, or about 17 percent of respondents, noted that they learned of the Manual specifically
when contacted with the survey or during the initial evaluation contact period. Of the other respondents,
they had learned about the Manual through visiting EPA's Web site, through their state contact, at
conferences or workshops, or from an EPA postcard. None had learned about it through an EPA
television or radio interview, and only one due to an EPA news release (see Figure below). All but one of
the respondents, or 95 percent, indicated that they were satisfied or very satisfied with their access to the
Manual; the other respondent was neither satisfied nor dissatisfied.
How Respondents Learned About the Manual
N=23
Don't Know/
Don't Remember
4%
Other
Correspondence
f-c-r E=A
9%
EPA news relesse
4%
November 2009
Page 10
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Other Findings
The evaluators researched where this Manual had been referenced on the Internet. Region 7 conducted
a number of outreach activities to announce the publication of the Manual, including developing a press
release. News aggregation and industry-focused Web sites and journals picked up this release and
developed short news stories for their readers, including Biofuels Journal4, Enviro Business and Legal
Reports5, Bulk Transporter6, and ChemAlliance7.
In addition, the Manual can be found on the Nebraska Ethanol Board's and Nebraska State Department
of Environmental Quality's Web sites8 9, and on the Web sites of some state departments that are not in
Region 7, including Kentucky10 and Minnesota.11 The Minnesota Web site noted that, while the Manual
"was not directly intended for Minnesota ethanol plants, many of the same policies and regulations apply."
The Manual had received 129,337 Web server requests ("hits") from its release in November 2007
through August 2009.
Topic Area II: Effectiveness of the Manual as an Information Tool
How effective is the Manual as a tool for transferring compliance (and related facility technical support)
information to ethanol facilities?
Overarching Findings
Participants in the evaluation found the Manual to be well organized, easy to navigate, and an appropriate
tool for conveying compliance information to facilities. Having this material, including appropriate
contacts, organized in one place provided for an easy reference for facility managers. Participants
indicated that they found the Manual to be readable and organized logically. Participants provided varied
and inconsistent feedback on whether the Manual was too technical or not technical enough, but
everyone interviewed and surveyed indicated that they found the Manual to be useful in some way.
Whether the Manual has the appropriate level of detail likely depends on the reader's existing level of
knowledge and facility needs.
Readable and Organized: Overall, respondents for this evaluation found the Manual to be readable
and well-organized.
4 "EPA Ethanol Manual: Environmental Compliance Manual for Ethanol Plants." Biofuels Journal. May/June 08,
page 44. Accessed on September 10, 2009: http://vwwv.nxtbook.com/nxtbooks/GJ/BJ_56_08/index.php
5 "EPA Region 7 Announces Rollout of Environmental Manual for Ethanol Facilities." Enviro Business and Legal
Reports. November 16, 2007. Accessed on August 19, 2009: http://enviro.blr.com/news.aspx?id=83282
6 "EPA develops ethanol manual for producers." Bulk Transporter, November 19, 2007. Accessed on August 19,
2009: http://bulktransporter.com/management/tank-truck/epa_ethanol_manual/.
7 "EPA ethanol Compliance Manual Available Online." ChemAlliance, September 5, 2008. Accessed on August 19,
2009: http://www.chemalliance.org/news/news_detail.asp?StorylD=2272
8 "Reports." Nebraska Department of Environmental Quality. Accessed on August 19, 2009:
http://www.deq.state.ne.us/Publica.nsf/Pages/06-192
9 "Resources." Nebraska Ethanol Board. Accessed on September 10, 2009: http://www.ne-
ethanol.org/resources/resources.htm
10 "BioFuel Considerations." Kentucky Department for Energy Development and Independence. Accessed on August
19, 2009: http://www.energy.ky.gov/dre3/renewable/considerations.htm
11 "Ethanol in Minnesota - information for industry." Minnesota Pollution Control Agency. Accessed on August 19,
2009: http://www.pca.state.mn.us/energy/fuels/ethanol-facilities.html
November 2009 Page 11
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
Level of Detail: Among respondents, there was disagreement over whether the document was too
technical or not technical enough, but everyone interviewed and surveyed indicated that they found
some use from the Manual. In addition, most respondents thought the right amount of information
was available in the Manual, though a few indicated that too little information was available and
provided ideas for where content could be expanded or added.
Reference Guide: As noted previously, almost all facilities who responded to this evaluation had
started production prior to publication of this Manual, so they were familiar with the material; however,
they expressed appreciation for having the information in one place to refer to when needed.
Contact Information: Respondents saw the value of having specific state and EPA contacts in
Appendix A of the Manual to refer to for more information, but noted that it would be more difficult to
keep up to date than other information in the Manual.
Interview Findings
"If anything you're trying to
reach a broad audience, and
the more broad you make it the
less technical it becomes... so
it doesn't necessarily have the
right level of detail for the finer
points in regulation for all
intended audiences." - EPA
contact
All interviewees provided an overall positive assessment of the
content and organization of the Manual, indicating that the Manual
was organized well and was very readable. Comments included
that it was helpful that the Manual was organized by EPA program
areas and that the language was very straight forward. Critiques of
the Manual from some respondents were that it may be slightly too
technical for those new to the field of environmental compliance and
permitting and that its length may prevent some potential users from
perusing it. However, no interviewees indicated that they saw
opportunities for shortening the Manual. Other interviewees,
however, indicated the oppositethat the Manual was too general
and needed more detail to be more useful to ethanol plants. One state interviewee noted that when a
facility first contacts them about an environmental permit, their state agency has a multi-media group that
provides information on the requirements for the facility; gives permit application forms and any other
relevant handouts; and offers to have an in-person meeting for any questions. This outreach provides
much of the same information found in the Manual, so the Manual itself was less useful in providing
unique information to this state's facilities.
All but one of the facilities interviewed had started up production prior to publication of this Manual, so the
general content of the Manual was already familiar to them. The one facility that started up after the
Manual was published used it as a reference guide for construction and referred to it occasionally during
operation. Other respondents noted that the Manual overall
was very useful, in that having all the information in one place
was in itself helpful. Another facility respondent said their
facility had put together a calendar based on the Manual to
help track due dates for permits and plans.
"There are differences between the
contacts in the appendix and the
contacts on the Iowa DNR Web site.
In some cases, the contact section
in the manual lists the head of the
department, which I probably
wouldn't need to contact. Before
using the contact information in the
manual I would check the Iowa DNR
Web site to make sure I contact the
appropriate person." - Ethanol
facility
The evaluators specifically asked all interviewees about the
"Who to Contact" section of Appendix A and whether this
section was useful and any ideas to improve it. Interviewees
indicated that the contact information in Appendix A was
helpful, though approximately half of the interviewees had not
looked at this section until asked about it during the interview.
A common comment on Appendix A was that it might be hard
for EPA to keep the contact information current and that
putting titles rather than contact names could help with this
November 2009
Page 12
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
challenge. Others, however, preferred to keep the specific contact names in the Appendix and have EPA
update the information frequently, perhaps on a linked Web site. One facility said that the contact
information provided in the Appendix did not always match the contact information found on their state
Web site or was not the appropriate level of staff person. One state interviewee said that he has received
calls from around the United States with questions about ethanol regulations and wished to have a list of
all state contacts for ethanol issues, rather than just those in Region 7.
Survey Findings
Survey respondents were generally positive about the Manual and its readability and organization.
Twenty of the 21 respondents, or more than 95 percent, who answered questions on readability
indicated that the Manual was somewhat easy or very easy to read (see Figure below).
Satisfaction with Manual's Readability
N = 21
47.6% 47.6o/0
0.0%__4.
Sixteen (76 percent) of respondents said it was easy or very easy to navigate to areas of interest,
while five (24 percent) said it was neither easy nor difficult to navigate the Manual. Twelve
respondents (almost 60 percent) thought there was the right amount of information in the Manual (see
Figure below).
Amount of Information in the Manual
N=21
Slightlytoo much
5%
Significantly too
much (0%)
Significantly too
little (0%)
No opinion (0%)
November 2009
Page 13
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
Topic Area III: Changes in Understanding and Behavior
Has the existence and use of the Manual led to increased understanding and behavior change to support
improved compliance?
Overarching Findings
This evaluation indicates that existence and use of the Manual has led to increased or enhanced
understanding and knowledge for ethanol facilities. However, this evaluation cannot conclusively say that
the Manual has led to any significant changes that would affect compliance on the part of facilities.
Understanding and Awareness: All of the facilities and contractors said the Manual was
comprehensive and was helpful for increasing or solidifying their understanding of environmental
compliance requirements.
Behavior Change: With only a few exceptions, respondents were not able to identify specific
instances where facilities had more awareness of applicable federal regulations or had made
compliance or management changes as a result of the Manual.
Interview Findings
"I used it because we were brand new,
and it was great for starting up. Now
that we're in production I refer back to
the Manual. I also belong to a group of
ethanol producers, and many times
people will call me with questions. I
use it as a reference tool to help
answer them." - Ethanol Facility
All of the facilities and contractors said the Manual was
comprehensive and was at least somewhat helpful for
increasing their understanding of environmental compliance
requirements, or solidifying what they already knew. The
contractors interviewed were all very familiar with the
applicable environmental rules, but all said that they thought
the Manual would be helpful to a facility starting out in the
construction and operation process. State interviewees
noted that if the ethanol industry rebounds, the Manual
would be very useful for helping new operators understand
requirements.
EPA and state compliance staff were asked whether facilities seemed more aware of applicable federal
environmental requirements as a result of the Manual. One EPA interviewee noted that there had been
increased awareness of the Risk Management Program (Clean Air Act, Section 112 (r)); this interviewee
said that, prior to the Manual, many plants did not realize they were covered under this regulation. Other
compliance staff noted that there may be increased awareness among the regulated community, but that
they could not conclusively point to the Manual as the cause of
that knowledge.
"It allowed us to not ship out work
to consultants; we did all the work
in house. In the past we would
have just hired a consultant, so I
believe it saved us money and
effort." - Ethanol Facility
Facilities, in general, could not point to instances where the use
of the Manual had influenced a compliance change. One
exception was a facility that had started up after the Manual was
published, who said, "Before we started up our plant, we read
the Manual and realized that we had missed a permit," which
prompted the facility to secure the permit. No facilities could
point to any environmental or health outcomes that resulted from
the use of the Manual. One facility noted that the main impact on their management processes was that
the information in the Manual allowed the facility to do more of their permitting work in-house rather than
hiring outside consultants.
November 2009
Page 14
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
One EPA interviewee recalled being contacted because of someone reviewing the Manual and having a
follow-up question.
Survey Findings
Overall, survey respondents expressed satisfaction with the information thoroughness of the Manual.
Sixteen respondents, or approximately 76 percent, were satisfied or very satisfied with how well the
Manual met their needs for understanding compliance information, and five, approximately 24 percent,
were neither satisfied nor dissatisfied; none expressed dissatisfaction. One contractor respondent was
dissatisfied with the information thoroughness of the Manual, suggesting that the air section should be
updated and expanded with more specific details. All other respondents were either satisfied or very
satisfied with the information thoroughness (15 respondents, or greater than 70 percent) or were neither
satisfied nor dissatisfied (five respondents, or 24 percent).
Compliance Data Analysis
Table 1. Pre- and Post-Manual Publication Ethanol Violations
Informal Violation or
Notice
Pre-Nov. 2007
48.7%
Informal Violation or
Notice
Post -Nov. 2007
34.6%
Formal Violation
Pre-Nov. 2007
11.5%
Formal Violation
Post-Nov. 2007
9.0%
N=78
Analysis of EPA's Enforcement and Compliance History Online (ECHO) database provided the above
information. ECHO provides an overview of a facility's environmental record under the Clean Air Act
(CAA) Stationary Source Program, the Clean Water Act (CWA) National Pollutant Elimination Discharge
System (NPDES), and the Resource Conservation and Recovery Act (RCRA), as well as state and
federal responses.
This information shows that there had been a decrease in informal violations since the Manual was
published in November 2007. Prior to November 2007, 48.7 percent of the facilities in Region 7 had
received an informal violation, compared to 34.6 percent after publication of the Manual. For formal
violations, 11.5 percent of Region 7 facilities received a formal violation before publication of the Manual,
compared to 9.0 percent after publication. However, this evaluation cannot conclusively state that
environmental compliance increased because of this Manual. The data in ECHO may have been
incomplete or lagging and may not have completely shown the enforcement actions taken in all of 2009,
and the evaluators did not have access to information on the rate at which inspections were conducted for
facilities in the Region on a yearly basis. In addition, this evaluation had incomplete information about
whether all of the plants analyzed were in full operation before November 2007 and whether they
continue to be in operation at the date of this evaluation. Further, past enforcement actions can influence
current compliance with regulations, as can other factorssuch as learning from the mistakes of other
facilities, a downturn or upturn in business, or changes in personnel with compliance responsibilities.
Finally, this evaluation did not have information about whether all of these facilities received and reviewed
the Manual and therefore what, if any, effect it may have had on their compliance behavior.
The evaluators also looked more specifically at the facilities that were interviewed for this evaluation and
that responded to the survey. Below are the findings of this analysis:
One facility had more than 15 informal violations between 2005 and 2007, but only two in 2008 and
2009, all pertaining to air regulations. The facility interviewee did not explicitly credit the Manual with
helping improve compliance, but did note that having the information in the Manual would have been
November 2009
Page 15
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
helpful when the facility was trying to be in compliance when just beginning operation, because the
Manual lays out all the regulations and needs clearly. The interviewee particularly noted that the
section on the CAA was helpful, as CAA requirements can be very confusing to facilities.
Another facility that responded to the survey had received 15 informal CAA violations between 2007
and 2009. In the survey, the respondent from this facility expressed satisfaction with the CAA
information in both the building and operation sections of the Manual. This respondent also
suggested that voluntary EPA inspections would be a helpful way to learn about compliance
information. This facility indicated that they had received the Manual from a state contact, though did
not indicate a timeframe for this interaction.
Another facility that responded to the survey had received a formal enforcement action under the
CAA earlier in 2009. This respondent expressed dissatisfaction with the information thoroughness of
the Manual and requested more specific examples and details.
As noted above, this evaluation cannot conclusively state that there is any causal link between these
compliance actions and use of the Manual. During discussions with the Region 7 Biofuels Team about
the findings of this report, members of the Biofuels Team noted that more consistent tracking of
compliance and beyond compliance behavior and targeting assistance based on more thorough
compliance information may be a path forward for this sector.
Topic Area IV: Improving the Manual
How could the Manual be improved to provide effective compliance assistance to ethanol facilities?
Overarching Findings
Respondents to this evaluation had a number of suggestions for how future versions of Manual could be
improved. The suggestions centered around additional content, such as a section on cellulosic ethanol,
or around functional content, such as checklists, case studies, call-out boxes, and examples.
Interview Findings
Most interviewees had some general comments and suggestions for how the Manual could be improved
to provide effective compliance assistance, as well as specific suggestions for additions. They offered the
following suggested changes and additions to the content:
A few contractors requested more examples be provided in the Manual to define terms, such as what
constitutes commencing construction and modifying a plant.
A contractor suggested including information on how long the permitting process takes, as well as a
general discussion of what permits would likely be required, as many operators start the process
without any knowledge of the expected permit timeframes.
A contractor suggested providing embedded hyperlinks to the Code of Federal Regulations (CFR)
Web site so that readers can look up the referring statutes.
A contractor suggested moving the Renewable Identification Numbers (RINs) requirements
discussion to a later section of the Manual, rather than up-front, since this requirement comes further
along in the facility operation process.
A contractor noted that some of the permits listed in the Manual did not seem applicable, and three
contractors noted other EPA permits and reporting plans applicable to ethanol plants that were not
November 2009 Page 16
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
mentioned in the Manual, including: Toxic Substance Control Act (TSCA) Form U Reporting and New
sources performance standards (Subpart VVa) (released after the Manual was initially published).
A contractor disagreed with the statement found in the Manual that "all ethanol plants will probably
generate some quantities of hazardous waste" (page 2-20), and noted that some facilities were
focusing on making use of their byproduct waste streams to make value-added products.
A facility suggested adding an executive summary to the front of the Manual to make the document
more approachable to potential readers.
In addition, the evaluators specifically asked interviewees about any suggested changes to the "Who to
Contact" section of Appendix A. There was no clear consensus from interviewees on whether this section
should provide direct numbers and names to contact people or to general support numbers. Interviewees
noted the benefit of having these specific names in knowing whom to contact but noted that staff turnover
could cause this section to be out of date sooner than other parts of the Manual. One state interviewee
suggested expanding this section to include all applicable ethanol offices in state agencies so that those
outside of Region 7 could use the Manual.
Interviewees were asked whether there were other areas of information that EPA should consider
covering in future versions of the Manual. The following were the suggestions provided for this area:
An EPA interviewee noted that issues have arisen in recent years around emissions from fermentors,
biochemical oxygen demand loading and storm water ponds, and leak detection. These issues and
others that have come to the attention of regulators may be candidates for specific discussions in
future versions of the Manual.
An EPA interviewee suggested that checklists would be helpful to facilities that have been in
operation and were familiar with much of the Manual content.
A state interviewee noted that the Manual was geared toward building a new plant, but that any future
version could include information about requirements when making substantial modifications to that
plant or process (e.g., using a new feed stock, burning fiber). Similarly, two contractors noted that
ethanol plants were looking for ways to integrate value-added products to their production stream
(e.g., oil extraction), and that including a discussion of requirements that plants should consider would
be useful. Related to this discussion, a state compliance staff member suggested a section on
byproduct handling.
An EPA interviewee suggested adding information about state laws to make the Manual more
comprehensive, while noting that it would be more difficult to update that information. Facilities and
contractors, similarly, said that ethanol plants were responsible for a number of federal requirements
to different agencies, including the U.S. Department of Transportation, the U.S. Department of
Homeland Security, the U.S. Army Corps of Engineers, and the Alcohol and Tobacco Tax and Trade
Bureau.
EPA and state staff as well as contractors noted that future versions could include sections on
cellulosic and other emerging types of ethanol.
State staff and contractors suggested that a future version of the Manual could include information on
what should happen with federal permits when plants halt production for any length of time. In
addition, unless there is renewed emphasis on alternative fuels, there may not be as much
construction of facilities in the future; any future versions could focus on operation rather than
construction of plants.
November 2009 Page 17
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Final Report
An EPA staff member suggested information on community and emergency response could be
valuable for this audience and that EPA should incorporate this information or link to another
organization that has resources in this area.
Survey Findings
The survey asked facilities and contractors what things EPA should consider to improve satisfaction with
the Manual as a compliance assistance tool. The following were the suggestions provided in the survey
for additional content or functionality:
More examples and specific details throughout the text
A list of ethanol relevant resources, such as ethanol trade associations
A list of contractors or consultants in particular states and the services they provide
Checklists and "at a glance" materials and pages
In addition, a facility expressed dissatisfaction with the transition between the construction and operation
sections of the Manual, but did not elaborate on this response. Another noted that the section on air
quality regulations needs updating and improving, but also did not expand on this comment.
The survey asked participants how frequently EPA should update the Manual to be helpful to facilities.
Ten respondents (48 percent) suggested updating the Manual every three years. The three respondents
who chose "Other" wrote in that Region 7 should update the Manual whenever there were regulatory
changes (see Figure below).
Which update frequency would be the most useful for the
Manual?
N=21
Other timeframe
14%
Every five years
9%
I don't intend to
refer to the
Manual again
(0%)
November 2009
Page 18
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Topic Area V: Other Compliance Assistance
What approaches other than the Manual would provide effective compliance assistance to the ethanol
facilities?
Overarching Findings
As previously discussed, evaluation participants thought the Manual was a useful tool for conveying
compliance information. However, evaluation participants also considered other venues, such as
conferences, EPA workshops, a more interactive Web site, and one-on-one interaction with ethanol
facilities to be additional valuable approaches for providing compliance assistance.
Interview Findings
During the interviews, facilities and contractors were asked about other tools and approaches that EPA
could use to present the information that is available in the Manual. Five of the ethanol facility
representatives noted that they had attended EPA workshops in the past and found them to be useful
sources of information about regulations and implementing them at their facilities. In addition, a few of
the contractors and facilities regularly attended conferences; additional or continued EPA presence would
be welcome there to share information about new resources or requirements. In addition, interviewees of
all categories noted that having an up-to-date Web site that is easy to navigate was a very effective way
to convey information. One contractor suggested having an ethanol-specific Web page, either nationally
or regionally, to provide links to resources and updates on regulations.
Survey Findings
The survey asked respondents about what other tools and approaches would be helpful for conveying
compliance assistance information. Most noted that compliance assistance had so far been sufficient.
Below were the other suggestions from the survey:
An email newsletter or listserv with reminders, updates, industry contacts, suggested best practices,
and regulatory changes applicable to the industry.
Voluntary EPA inspections without fines.
Additional workshops and meetings with open discussions between regulators and ethanol plants.
November 2009 Page 19
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
IV. Recommendations
This evaluation assessed the effectiveness of an ethanol compliance assistance Manual developed by
EPA Region 7. The evaluation focused on a set of overarching evaluation questions to inform this
assessment. Evaluation participants clearly conveyed that the Manual was useful and valuable to its
intended audiences, and that it effectively communicates relevant information for ethanol plants. The
following recommendations focus on ways to improve the Manual for any potential future versions and
other ways that Region 7 could make this information available to its intended audiences.
The evaluators understand that there may be limited resources available for updating the Manual and
limited time available for staff to provide other kinds of compliance assistance to ethanol facilities. The
evaluators also recognize that the market for ethanol has been shifting dramatically, and that the demand
for new facility permits has not returned to the peak it was in when Region 7 first released the Manual.
These and other related factors will likely influence demand for the Manual and feasibility of implementing
any of the evaluation's recommendations. The below recommendations are organized thematically and
not by priority.
1. As a first step, consider whether to tailor the Manual's content and distribution to the national
level.
The Manual is available to the public on the Region 7 Web site, and as such it is available nationally.
However, the contact information in Appendix A is focused on Region 7 only, and EPA's national offices
have not posted the Manual on their Web sites. EPA and state compliance staff and contractors noted
that the Manual's content was not specific to Region 7, and would be of interest to other Regions with
significant ethanol industries. Already, some state agencies such as the Kentucky Department for Energy
Development and Independence and the Minnesota Pollution Control Agency have linked to the Manual
on their Web sites, noting that it may be of interest to their states' ethanol facilities.
The Region 7 Biofuels Team noted that they had written the Manual in such a way that it could be easily
adapted to other EPA Regions. The Region 7-specific contact information, for example, was included in
an appendix to enable easy substitutions with other Regions' contact information. Sending the Manual to
EPA staff in other Regions for posting and distribution would be a simple first step to making this
information more widely available, and Regional staff or Headquarters could take the lead on revising the
contact Appendix and any other sections that needed another Region's specific information. Staff in the
other Regions and at state environmental agencies would likely need to be available to answer questions
about the information provided by the Manual, which would require a commitment of staff time in other
Regions with significant numbers of ethanol plants.
The evaluators recognize that there may not be a clear national EPA office or program to take over
responsibility for the current Manual, any updates to the Manual, or any coordination between or among
Regions for the Manual. Similarly, other Regions may not have an equivalent biofuels team or group to
coordinate the Manual's distribution or incorporate regional contact information. For these reasons, if the
Manual were to be used nationally or in other Regions, it may be necessary for the Biofuels Team in
Region 7 to identify individual "champions" for the Manual's use and updates at the national and/or
regional levels. Some level of ongoing coordination between Regions or between Regions and
Headquarters may be needed.
November 2009 Page 20
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
2. Consider additional outreach strategies and targeted audiences.
Region 7 undertook a number of activities to advertise the Manual's existence, including a news release,
providing postcards at conferences, emailing those who had expressed interest, and highlighting the
Manual during media interviews. The Region could continue to advertise the Manual and any future
versions with these mechanisms, but could also expand its outreach to consider additional audiences and
outreach strategies.
The evaluators recognize that Region 7 had limited resources to conduct Manual outreach and
advertisement. Allocating at least some resources to outreach would allow the Region to conduct
activities such as advertising in trade journals relevant to the ethanol industry. However, Region 7 could
also undertake activities that would not require significant resource investment and would be more likely
to reach the targeted audiences relatively quickly:
Region 7 could develop and maintain an ethanol-industry email listserv that regularly sends out
updates and news to subscribers. Region 7 could allow anyone to join or have a moderated listserv
of only ethanol plant managers. This approach would be a simple way to keep in contact with
interested stakeholders and alert them to any future versions or updated Manual, or to remind them of
the Manual's existence, as well as send out other relevant information.
When Region 7 staff members visit ethanol plants on site visits, the staff could bring an electronic
copy of the Manual or could provide a postcard with the Web link to the Manual.
If the Manual is updated with new information, Region 7 could gather a small focus group of three to
five ethanol facility contacts by phone or in-person. This group could discuss the best ways to inform
ethanol plants about any changes, and whether the industry needs a fully updated Manual with the
information, or whether an email or Web site with the changes is sufficient. This approach would
allow Region 7 to learn about any venues (e.g., upcoming conferences) that could be key distribution
points, and would also allow Region 7 to target its resources strategically.
While ethanol plants were the primary targeted audience for this Manual, ethanol-industry contractors
could be an effective audience for future outreach. Many of the facilities interviewed said that they use
contractors for much of their environmental compliance and permitting work, and rely on these contractors
for updates and information on regulation. As there are a smaller number of contractors and consultants
working with existing and potential ethanol facilities, making the Manual specifically available to
contractors could more directly reach an important audience and perhaps ultimately reach a wider
audience. In addition, having contractors review draft Manual updates may be a productive way to
ensure that the Manual is helpful to the target audience.
3. Consider making this tool primarily Web-based, with a hard copy Manual as a supplement.
Nearly all evaluation participants accessed the Manual on the Internet, and many suggested that the
Manual include more Web-based content, such as updates and additional or more current contact
information. Transferring the Manual to a current, Web-based version would require some effort,
particularly if user interactivity such as topic "drill downs" and links to related topics or contacts, were to
be incorporated. Making the content Web-based, however, would make the Manual more accessible and
useful to many users and likely easier to maintain and update by EPA. In addition, Internet search
engines could find specific sections more easily, and Region 7 could implement Web feeds that would
allow users to subscribe and be notified of any changes.12 A version formatted for printing could still be
12 "Web feeds" are a format for providing users with frequently updated content. Content distributors (in this case,
Region 7) develop a "feed" and allow users to subscribe to it. Users are alerted through a "feed reader" or through
November 2009 Page 21
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
available on the Web for readers who wanted to download it for their office reference library or for EPA
and state compliance staff or others who would want to take hard copies to facilities they visit.
The below are some ways that Region 7 could implement this recommendation:
Region 7 could simply upload the content onto a series of HTML Web pages. This approach would
allow EPA staff to make changes quickly to the content of the Manual, without republishing the
document entirely. Examples of this approach at EPA include the EPA Lean in Government Starter
Guide13 and the EPA Guide for Industrial Waste Management.14
Region 7 could upload the Manual to the Web in a wiki-format,15 which would allow EPA staff to edit
the content even more easily than HTML pages, and could allow other non-EPA users to edit the
content. This ease of updating can help ensure that content is up-to-date and relevant to its intended
audiences. Region 7 could upload the content to an EPA-hosted or third-party hosted wiki and then
designate through assigned log-ins which groups (e.g., EPA staff, state compliance staff, facility
contacts) are allowed to contribute edits to the document and create discussions about specific pages
or sections. This approach could allow a greater cross-section of stakeholders to contribute to
improving the Manual, and could result in a more valuable resource for ethanol plants looking for
compliance assistance information. If Region 7 decides to go this route to make the Manual Web-
based, they can look to the lessons learned of similar efforts at EPA and other federal agencies. For
example, EPA's Watershed Central Wiki is a Web site based on the MediaWiki platform that is open
to EPA employees and identified partners to provide information on watershed resource management
best practices, organizations, and plans.16
Region 7 could develop a Web site that displays the content of the Manual on a series of HTML or
Wiki Web pages (discussed above), and also links to more dynamic content and resources internal
and external to EPA, such as upcoming meetings or relevant conferences. For example, the EPA
Office of Wastewater Management has developed a Web site on NPDES (National Pollutant
Discharge Elimination System) permits.17 It links not only to compliance information, but also to
relevant new publications and training materials, discussions of current issues, and links to upcoming
meetings and conferences. While incorporating and updating this material would require more staff
time and resources on a continuous basis, it could become a frequently used and valuable resource
for ethanol plants and their environmental managers.
4. Update with current information on regulations.
If a future version of the Manual is published, it could incorporate the latest information on applicable
federal environmental regulations, permits, plans, and approvals for ethanol plants. Alternatively, a
stand-alone supplement to the Manual that focuses solely on recent changes and updates could be
created. If the Manual was Web-based (see Recommendation 3), regular maintenance would ensure that
all of these components are kept current. If the Manual continued to be hard-copy based, then having
their email when new content is available. An example of this is EPA's Office of Public Affairs news release pages,
which alerts users whenever a new press release is issued; see http://www.epa.gov/newsroom/rssfeeds.htm.
13See http://www.epa.gov/lean/starterkit/index.htm
14See http://www.epa.gov/waste/nonhaz/industrial/guide/index.htm
15 A wiki is a Web site that allows easy creation and editing of content for users. Wikis are often used for
collaborative editing of a document that has content that changes frequently and/or could benefit from multiple
perspectives. Wikipedia (http://www.wikipedia.org) is one of the best known wikis.
16 See https://wiki.epa.gov/watershed/index.php/Main_Page; Access is limited to EPA employees and identified
partners.
17 See http://cfpub.epa.gov/npdes/index.cfm
November 2009 Page 22
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
links within the Manual to an EPA Web page that provides real-time updates to the Manual content would
keep the information current for ethanol facilities and contractors looking for updates.
5. Create an executive summary to encapsulate the main points of the Manual.
A few participants in this evaluation noted that the length of the Manual could detract potential readers
from reviewing the Manual; however, no participants noted opportunities for shortening the Manual.18 In
addition, several participants thought that a short executive summary or equivalent synopsis would give
readers an overview of the content and any potential sections of interest without taking away any of the
content within the Manual. There is currently a front section ("What is the purpose of this manual?") that
provides two paragraphs on the purpose of the Manual and some disclaimers on what the Manual does
not cover; however, this section is not an executive summary in the way that readers expect. Similarly,
the Introduction covers three topics (requirements for renewable fuels, the renewable fuel standard
program, and a brief overview of the ethanol production process), but does not include a list of the laws
that apply when constructing, modifying, or operating ethanol plants or any related summary information.
Creating a new executive summary or overview section to summarize the main topics of each of the
chapters could be one way to meet the stated need of respondents.
6. Incorporate more "at a glance" information.
Expanded "at a glance" information, such as lists of resources, call-out boxes, checklists (e.g., on the
steps to take), and calendars can be integrated throughout the Manual to highlight particularly important
information. Respondents to this evaluation indicated that they would like more of this kind of content so
that they can quickly refer to information of interest. The evaluators recognized that the Biofuels Team
intentionally avoided creating new guidance with the Manual, and as such that there may be reason not to
create action or requirements lists that could constitute guidance. If this has not been done already, it
may be worth exploring the feasibility of drawing from existing guidances to develop "how to" checklists or
other short reference lists either at the regulation, media, and/or cross-media levels.
Similarly, a generic schedule that a plant could modify and download could be very useful. Such a
schedule could, for example, indicate a starting date and include the deadlines (e.g., date + 90 days) for
the related required procedures, submissions, and other information. It may be that plants or their
contractors already have these tools in paper or electronic format and may be willing to share them for
use by EPA for this purpose. As mentioned earlier in this report, one facility interviewee developed a
calendar for this purpose after reading the Manual.
Several participants raised the idea of listing resources beyond the contacts already included in Appendix
A. Some said that a list of contractors would be of interest, though they also said that caution may need
to be exercised in doing so to avoid perceived or actual conflicts of interest. Some thought that there
were precedents for listing contractors and that there would be no problems in doing so as long as there
were multiple contractors listed. Others thought that directing parties to external Web sites that already
list contractors would be a relatively easy way to achieve this change, though it is not clear that there is
such a resource. Other resources that Region 7 could consider could be lists of facilities that would be
willing to discuss their experience with others and contacts from other regulatory agencies, such as the
Occupational Safety and Health Administration (OSHA).
18 One way to reduce the manual's size would be to reformat it with fewer pictures (though the pictures make it a
much more visually pleasing document), reduce the large margins, and modify the spacing. Converting the Manual
to a Web-based format may or may not assist with length depending on how the Web site is designed. This would
not serve as a substitute for including an executive summary but would likely help with the initial impression voiced by
several participants that the Manual seems very long.
November 2009 Page 23
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
In developing the Manual, the Biofuels Team has considered about whether a "flip book" of key issues
and important points would be feasible. The evaluators think that anything along these lines, regardless
of the format (e.g., flip book, standard paper format, electronic spreadsheet) would be useful for readers.
7. Provide more specific examples and case studies.
Participants commented that the information in the Manual was sometimes vague or too general and that
specific examples would be helpful for understanding. For example, providing examples of what
constitutes a modification to a building would be helpful to a reader wanting to understand what is
applicable to their building. More narrative discussion or case studies, either real or hypothetical, can
also help a reader relate their facility to the regulations discussed within the Manual.
8. Expand content to include regulatory information about topics of interest to the target
audience, such as cellulosic ethanol production and byproduct diversion into other products.
There were many suggestions for specific content to add to the Manual for future versions (see the
findings on Topic Area IV: Improving the Manual). Cellulosic ethanol production and byproduct diversion
into other products were mentioned most often by the state compliance staff, facilities, and contractors
who participated in the evaluation. Cellulosic ethanol production is expected to increase as producers
look to different feedstocks to take advantage of new incentives, research findings and changing market
conditions, and facilities are interested in the different federal environmental regulations that may be
applicable. In addition, during this evaluation, ethanol facilities indicated that they were looking for ways
to take the byproducts of ethanol production, make them into value-added products, increase profit
margins, and decrease waste disposal. Facilities may find value in a discussion in the Manual of
regulations and requirements to consider when changing operations in these ways.
9. Expand information on beyond compliance efforts and best practices.
Although the Manual is focused on current federal environmental regulations and rules, a section on how
a facility could go beyond compliance and implement best practices may be of interest to some facilities.
In particular, newer or smaller facilities may not know about best practices that could provide
environmental, safety and/or economic benefits; having this information in one place could be useful.
Currently, the Manual has a section in the construction chapter about the Pollution Prevention Action and
refers to the EPA Pollution Prevention Web site.19 Region 7 could evaluate whether this is sufficient
information or whether expanding this section with additional best practices could be a way to incorporate
more beyond compliance information for the audience. The evaluators recognize, however, that
providing beyond compliance information is a secondary priority following the Manual's focus on
compliance assistance.
10. Apply the lessons from this evaluation to the Region's other compliance assistance and
beyond compliance efforts.
The evaluators understand that Region 7's compliance assistance activities are largely integrated into its
broader compliance-enforcement program. The Region has a compliance assistance coordinator who
serves the regional media program staff as a policy disseminator and a data quality control check for
measuring activities. The technical and enforcement staff in the other regional programs incorporate the
actual compliance assistance activities into their other routine actions. More broadly, the Region focuses
on conducting inspections, following up on potential violations, and, in general, working within the
traditional compliance-enforcement framework that has been the hallmark of EPA's compliance approach
for over 30 years. Region 7, like other Regions, also oversees the compliance enforcement work of the
1Q
See http://www.epa.gov/p2/
November 2009 Page 24
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
Region 7 states and reports back to EPA Headquarters Office of Enforcement and Compliance
Assurance (OECA) and on its activities and progress through routine reporting channels. Compliance
assistance is, in other words, primarily an "add on" to more "core" programmatic activities and
responsibilities.
Within this broader construct, the Region conducts both ad-hoc and routine periodic compliance
assistance work. The Manual is an ad-hoc example that the Region took on after a dramatic shift in the
number of new permit requests for ethanol facilities as explained in this report. In another instance,
Region 7 is focusing on improving compliance with a new lead (Pb) rule, which was promulgated in April
2008. This rule requires workers to follow lead-safe work practice standards to reduce potential exposure
to dangerous levels of lead during renovation and repair activities.20 The Region has been conducting
outreach with thousands of contractors and citizens who renovate, repair, or repaint older buildings that
may have lead-based paint to support compliance with the new rule. The Region also discusses
compliance assistance activities each quarter with the Region 7 states. At these meetings, the state and
regional staff take turns sharing their compliance assistance activities among discussions of other
activities for collaboration.
Within this context, the evaluators, in consultation with Region 7 staff, have identified some options for
applying the lessons learned from the evaluation of the ethanol compliance assistance manual to the
Region's broader compliance assistance activities. These options, which can serve as the basis for
further discussion within the Region, are as follows:
i. Determine whether the Region would like to initiate more compliance assistance activities like the
ethanol compliance assistance manual that are proactive in nature and intended to help raise
awareness amongst a targeted regulated community that has been identified as needed additional
support. Doing so would require additional planning and coordination and therefore the "blessing" of
senior managers who would deploy staff resources for this purpose. In addition, there may need to
be further exploration of the relationship between the compliance assistance work and the more
customary interactions that compliance inspectors have with the regulated community to clarify
respective roles and responsibilities.
ii. Consider options for cross-media compliance assistance integration like the Biofuels' Team effort on
the ethanol manual. Evaluation participants expressed their appreciation for having all major
regulations described in one place instead of media-by-media and program-by-program.
iii. Consider additional opportunities for coordinating with, as well as learning from, the Region 7 state
compliance programs. Although Region 7 already works hand-in-hand with the state compliance
enforcement programs, there are likely opportunities to be more proactive in terms of identifying
potential compliance gaps or assistance opportunities, learning about planned or ongoing compliance
activities that the states are conducting and coordinating on compliance assistance implementation.
There are likely efficiencies and economies of scale for doing so; for instance, in providing regional
compliance assistance materials (such as the Manual) to state compliance staff and vice versa, or in
having strategic planning meetings intended to design and coordinate Region-wide compliance
activities for the upcoming years (this would be materially different from the current update-oriented
quarterly meetings).
iv. Take advantage of Internet-based tools and options. Tools such as Webinars, interactive guides,
Wikis, and topical drill-down Web sites (such as the one discussed in the context of the ethanol
20 See: http://www.epa.gov/region7/citizens/lead.htm
November 2009 Page 25
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report
compliance assistance manual) provide opportunities for getting compliance information out to a wide
range of people quickly.
In general, these ideas and options will depend on whether Region 7 wishes to invest in more targeted
compliance assistance on top of the current compliance-enforcement activities. There are likely to be
needs related to existing or new rules where additional compliance assistance focus would help to raise
awareness and support compliance before any inspectors contact regulated entities. Perhaps Region 7
is already exploring such options in relation to other sector outreach, or perhaps this is an area that is
worth exploring.
As a first step, Region 7 may wish to identify a point person or small group of individuals to further explore
the options for either following up on recommendations related to the ethanol compliance assistance
manual itself, or on taking a next step on additional targeted compliance assistance that can benefit from
the lessons gained from the Manual evaluation. It is likely that a targeted scoping exercise could result in
a short set of viable options and recommendations for future "value added" compliance assistance that
Region 7, its states, and the Regional regulated community could all utilize to support improved
understanding of and compliance with the law.
November 2009 Page 26
-------
?/ERA
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
November 2009
EPA-100-R-10-001
Evaluation of the
Effectiveness of an
Ethanol Compliance
Manual
Report Appendices
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Table of Contents
Appendix A. Customer Satisfaction Survey 1
Appendix B. Customer Satisfaction Survey Results 10
Appendix C. Interview Guide for Ethanol Facilities 19
Appendix D. Interview Guide for Ethanol Contractors 22
Appendix E. Interview Guide for State Compliance Staff 24
Appendix F. Interview Guide for EPA Staff 26
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
EPAICRNo. 1711.05
OMB Control No. 2090-0019
EXPIRATION DATE: 8/31/2009
Appendix A. Customer Satisfaction Survey
COMPLIANCE ASSISTANCE MANUAL
FOR ETHANOL FACILITIES
Survey of Customer Satisfaction
Thank you for participating in this survey so that EPA can serve you more effectively.
Assuming you have already read the compliance assistance manual, Environmental Laws Applicable to
Construction and Operation of Ethanol Plants, we anticipate that the main part of this survey will take
approximately 10-20 minutes and the optional questions at the end will take approximately an additional
10 minutes. Your participation will help EPA improve ethanol facilities' satisfaction with this manual.
You may also take this survey on-line at the following website:
http://www.survevmonkey.eom/s.aspx?sm=2uWpdtMi8JSWiH 2bkMbQ35g 3d 3d.
Submitting your responses online will take the same amount of time and will help us to analyze the
results and provide feedback to EPA more quickly.
If you choose to fill out the paper survey instead, please return
your responses by July 10, 2009 in the enclosed self-addressed
stamped envelope.
Note about respondent confidentiality: The contractors evaluating
the ethanol facilities' and contractors' satisfaction with the compliance
assistance manual, Environmental Laws Applicable to Construction and
Operation of Ethanol Plants, are gathering basic information to better
understand how useful different types of customers have found the
Manual to be. Informational responses will be generalized and made
anonymous for the final results and no facility or contractor names or
other identifiers will be provided to EPA as part of the results, unless a
facility or contractor explicitly indicates at the end of this survey that
they would like their responses to be identified.
'-.' I ;:M|, M | M ',( LAWS
Ethanol Plants
T» m-a-CKH.'!
jV .\ ±^!L
.'->,%
Industrial Economics, Incorporated (http://vwvw.indecon.com/) and Ross & Associates Environmental Consulting,
Ltd. (http://www.ross-assoc.com) are the contractor team. For more information, please contact Anna Williams
(anna.williams@.ross-assoc.com) or Amy Wheeless (amv.wheeless@.ross-assoc.com) by email or at (206) 447-
1805.
November 2009
Page 1
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
I. PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER
1. Prior to receiving this questionnaire, were you aware that EPA developed the Compliance
Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available?
o
Yes
O
No
2. Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities,
published by EPA in November 2007? If No, please skip to Question 17.
O
Yes
O
No
3. Please check the method below that indicates how you learned about the Manual
O
Postcard
distributed from
EPA
0
Meetings /
conferences (e.g.,
O
EPA news release
0
Other
correspondence
O
EPA radio or TV
interview
0
Other method
(please specify)
O O
EPA Web site State contact
0
Don't know/don't
remember
FEW workshop, from EPA
CenSARA's National
Environmental
Biofuels
Conference)
If other:
4. Why were you interested in receiving the Manual? (check all that apply)
00 00
Wanted to learn more Wanted to learn about General reference Other
about applicable Federal applicable Federal laws for
laws for construction of operation of facilities
facilities
If other:
November 2009
Page 2
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Optional response: O I received the Manual without asking for it (check here if applicable)
PART 2: ACCESSING THE MANUAL
5. Please check the method below that indicates how you accessed the Manual.
O
O
O
O
O
Downloaded the Received a Received a CD
Manual from the Compact Disc (CD) from a state
EPA Web site from EPA contact
Received a CD Received a hard
from another copy from EPA
contact outside of
my facility
O
O
O
Received a hard Received a hard Other method
copy from a state copy from a (please specify)
contact different (not
state/EPA) contact
outside of my
facility
If other:
6. Please rate how satisfied you were with the ease of accessing the Manual.
000000
Very Dissatisfied Neither Satisfied Very Satisfied No Opinion
Dissatisfied Dissatisfied nor
Satisfied
Please describe any ways you think EPA could improve access to the Manual (optional):
7. Have you read any part of the Manual? If No, please skip to Question 17. If Yes, please continue
to Part 3 of the survey.
O O
Yes No
November 2009
Page 3
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
PART 3: STRUCTURE AND READABILITY OF THE MANUAL
8. Please rate your satisfaction with the Manual's readability.
0000
Very difficult to Difficult to read Neither easy
read nor difficult to
read
o
Somewhat easy Very easy to
to read read
o
No opinion
9. Please rate the ease of navigating through the Manual to get to areas of interest.
0
Very difficult to
navigate to
areas of interest
0
Difficult to
navigate to
areas of
interest
10. How would characterize the
O
Significantly too
little
O
Slightly too
little
0
Not easy, but
not difficult to
navigate to
areas of
interest
0
Easy to
navigate to
areas of
interest
amount of information presented
O
Right Amount
O
Slightly too
much
0
Very easy to
navigate to
areas of
interest
in the Manual?
O
Significantly
too much
0
No opinion
O
No opinion
We welcome any comments on your response:
IV. PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL
11. a. Please rate your satisfaction with how well the Manual met your needs for understanding
compliance information.
Very Dissatisfied Dissatisfied
O O
Neither
Dissatisfied nor
Satisfied
O
Satisfied
O
Very
Satisfied
O
No Opinion
O
November 2009
Page 4
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
b. Please rate your satisfaction with how clear the information provided in the Manual was on
where to go for more information.
Neither
Dissatisfied nor Very
Very Dissatisfied Dissatisfied Satisfied Satisfied Satisfied No Opinion
000 000
c. Please rate your satisfaction with the information thoroughness of the Manual.
Neither
Dissatisfied nor Very
Very Dissatisfied Dissatisfied Satisfied Satisfied Satisfied No Opinion
o o o o o o
We welcome any comments on your response:
Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If so,
we welcome any explanation and suggestions for improving your satisfaction with that section or
chapter below.
OYes ONO
12. What else should EPA consider to improve your satisfaction with the Manual as a compliance
assistance tool?
November 2009 Page 5
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
13. Please rate your overall satisfaction with the Manual as a compliance assistance tool.
Very Dissatisfied Dissatisfied
o o
Neither
Dissatisfied nor
Satisfied
O
Very
Satisfied Satisfied No Opinion
O O O
V. PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE
14. If EPA were to update the Manual, which update frequency would be the most useful?
O
Annually
If other:
O
Every 3 years
O
Every five years
O
Other timeframe
(please specify)
O
I don't intend to
refer to the
Manual again
15. What other tools and approaches would you prefer for sharing compliance assistance
information concerning ethanol facilities?
VI. FINAL QUESTIONS!
16. What category best describes your position?
O O
Facility manager Facility compliance officer
O
o
Other facility role Contractor
(please specify)
If other:
17. Can we identify your facility or company (by name) as having participated in this survey?
OYes ONO
November 2009
Page 6
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Facility or Company Name (optional)
PRIMARY POINT PERSON (OPTIONAL)
If you are willing to provide the name of a primary point person who could be contacted by the
contractors with any follow up questions or correspondence, please list that person here.
Name:
Phone Number:
E-mail Address:
Thank you for participating in this survey!
Please return your responses using the enclosed self-addressed stamped envelope by
July 10, 2009.
November 2009 Page 7
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Optional Section: if you would like to provide more specific feedback on each Section of the
Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer
the following questions.
1. Please rank your overall satisfaction with the ease of understanding, information thoroughness,
and overall usability of each of the Manual's sections:
Neither
Dissatisfied
Very nor Very No
Dissatisfied Dissatisfied Satisfied Satisfied Satisfied Opinion
Introduction
Section 1.1: Requirement Q O O O O O
for Renewable Fuels
Section 1.2: Renewable Q O O O O O
Fuel Standard Program
Section 1.3: Brief
Overview of Ethanol O O O O O O
Production Process
Chapter 1: What Laws Apply When I'm Constructing or Modifying a Plant
Section 1.1: National
Environmental Policy Act
Section 1.2: Clean Air Act
Section 1.3: Clean Water
Act
Section 1.4: Safe
Drinking Water Act
Section 1.5: Pollution
Prevention Act
Chapter 2: What Laws Apply
Section 2.1: Clean Air Act
Section 2.2: Emergency
Planning and Community
Right to Know Act
Section 2.3: Clean Water
Act
Section 2.4: Safe
Drinking Water Act
O
O
O
O
O
to Operating
O
O
O
O
O
O
O
O
O
an Ethanol
O
O
O
O
O
O
O
O
O
Plant?
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
November 2009 Page 8
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Neither
Dissatisfied
Very nor Very No
Dissatisfied Dissatisfied Satisfied Satisfied Satisfied Opinion
Section 2.5: Resource
Conservation and
Recovery Act -
Hazardous Waste
Appendices
Appendix A (First Half):
Summary of Laws Pertaining
to Ethanol Production and
Contacts
Appendix A (Second Half):
Contact Information
Appendix B: Emergency
Planning
Appendix C: Clean Air Act
Section 112-R Requirements
Appendix D: Definitions Of
Acronyms
Appendix E: The NEPA Process
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
Do you have any specific suggestions for improving the usefulness of any of the Manual sections?
(optional)
If you have not done so already, please remember to fill out the final section of the main part of the
survey starting with "Part 4: Your Overall Satisfaction with the Manual" on page 3. Thank you!
Burden Statement: Public reporting burden for this collection of information is estimated to average fifteen (15)
minutes per response, including the time for reviewing instructions, gathering information, and completing and
reviewing the collection of information, with a possible ten (10) additional minutes for the optional section at the
end. Send comments on the Agency's need for this information, the accuracy of the provided burden estimates,
and any suggestions for reducing the burden, including the use of automated collection techniques to the Director,
OEI Collection Strategies Division, United States Environmental Protection Agency (Mail Code 2822T), 1200
Pennsylvania Avenue, Washington, DC 20460; and to the Office of Information and Regulatory Affairs, Office of
Management and Budget, 725 17th Street NW, Washington, DC 20503, Attention: Desk Officer for EPA. Include the
EPA ICR number (1711.04) and the OMB control number (2090-0019) in any correspondence.
November 2009
Page 9
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Appendix B. Customer Satisfaction Survey Results
Note: Questions that asked for optional contact information are not listed here, nor are any participant
comments that indicated which surveys responded to the survey.
PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER
1. Prior to receiving this questionnaire, were you aware that EPA developed the Compliance
Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available?
Response Response
Percent Count
Yes 72.7% 16
No 27.3% 6
answered question 22
skipped question 2
2. Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities,
published by EPA in November 2007? If No, please skip to Question 17.
Response Response
Percent Count
Yes 100.0% 22
No 0.0% 0
answered guest/on 22
skipped question 2
3. Please check the method below that indicates how you learned about the Manual
Response Response
Percent Count
Postcard distributed from EPA 13.0% 3
EPA news release 4.3% 1
EPA radio or TV interview 0.0% 0
EPA Web site 21.7% 5
State contact 17.4% 4
Meetings / conferences (e.g., FEW workshop,
CenSARA's National Environmental Biofuels 13.0% 3
Conference)
Other correspondence from EPA 8.7% 2
Other method (please specify) 17.4% 4
Don't know/don't remember 4.3% 1
Other (please specify)
From corporate office
From this survey and this consulting firm 4
This questionnaire
From Ross & Associates
November 2009 Page 10
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
answered question
skipped question
23
1
4. Why were you interested in receiving the Manual? (check all that apply)
Response
Percent
Wanted to learn more about applicable Federal laws
for construction of facilities
Wanted to learn about applicable Federal laws for
operation of facilities
General reference
Other (please specify)
I received the Manual without asking for it
answered question
skipped question
22.7%
63.6%
54.5%
4.5%
Response
Count
14
12
1
3
22
2
PART 2: ACCESSING THE MANUAL
5. Please check the method below that indicates how you accessed the Manual.
Downloaded the Manual from the EPA Web site
Received a Compact Disc (CD) from EPA
Received a CD from a state contact
Received a CD from another contact outside of my
facility
Received a hard copy from EPA
Received a hard copy from a state contact
Received a hard copy from a different (not
state/EPA) contact outside of my facility
Other method (please specify:)
Received electronic copy from corporate office
Electronic copy from state contact
Received link from Ross & Associates
Emailed to me by EPA contact
answered question
skipped question
Response
Percent
73.7%
0.0%
0.0%
Response
Count
14
0
0
0.0%
15.8%
5.3%
5.3%
3
1
19
5
6. Please rate how satisfied you were with the ease of accessing the Manual.
Very Dissatisfied
Dissatisfied
Neither Dissatisfied nor Satisfied
Satisfied
Very Satisfied
Response
Percent
0.0%
0.0%
4.3%
43.5%
39.1%
Response
Count
0
0
1
10
9
November 2009
Page 11
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
No Opinion
^^ 13.0%
answered question
skipped question
23
1
Please describe any ways you think EPA could improve access to the Manual (optional):
No Answers
1. Have you read any part of the Manual? If No, please skip to Question 17. If Yes, please continue
to Part 3 of the survey.
Response Response
Percent Count
91.3% 21
Yes
No
^^ 8.7%
answered question
skipped question
23
1
PART 3: STRUCTURE AND READABILITY OF THE MANUAL
8. Please rate your satisfaction with the Manual's readability.
Very difficult to read
Difficult to read
Neither easy nor difficult to read
Somewhat easy to read
Very easy to read
No opinion
9. Please rate the ease of navigating through the Manual to get to areas of interest.
Very difficult to navigate to areas of interest
Difficult to navigate to areas of interest
Not easy, but not difficult to navigate to areas of
interest
Easy to navigate to areas of interest
Very easy to navigate to areas of interest
No opinion
Response
Percent
0.0%
0.0%
4.8%
47.6%
47.6%
0.0%
answered question
skipped question
Response
Count
0
0
1
10
10
0
21
3
Response
Percent
0.0%
0.0%
Response
Count
0
0
23.8%
57.1%
19.0%
0.0%
answered question
skipped question
12
4
0
21
3
November 2009
Page 12
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
10. How would characterize the amount of information presented in the Manual?
Significantly too little
Slightly too little
Right Amount
Slightly too much
Significantly too much
No opinion
Response
Percent
0.0%
38.1%
57.1%
4.8%
0.0%
0.0%
answered question
skipped question
Response
Count
0
8
12
1
0
0
21
3
We welcome any comments on your response:
No Answers
I IV. PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL 1
11. a. Please rate your satisfaction
How well the Manual
met your needs for
understanding
compliance information
How clear the
information provided in
the Manual was on
where to go for more
information
The information
thoroughness of the
Manual
Very
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
with the following:
Neither
Dissatisfied Di^atisfied
Satisfied
0 (0%) 5 (23.8%)
0 (0%) 6 (28.6%)
1 (4.8%) 5 (23.8%)
Satisfied
13
(61.9%)
1 1
j. j.
(52.4%)
13
(61.9%)
Very
Satisfied
3
(14.3%)
4
I
(19.0%)
2 (9.5%)
No
Opinion
0 (0%)
0 (0%)
0 (0%)
answered quest/on 21
skipped question 3
We welcome any comments on your response
Should have provided some more details.
November 2009
Page 13
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If
so, we welcome any explanation and suggestions for improving your satisfaction with that
section or chapter below.
Response
Percent
Yes 9.5%
No 90.5%
answered question
skipped question
Response
Count
2
19
21
3
We welcome any comments on your response:
The section on air quality regulations needs improvement and updating.
Construction to operations transition
What else should EPA consider to improve your satisfaction with the Manual as a compliance
assistance tool?
A section targeted to the regulators (EPA/states) where the ethanol plants can provide
input on details of plant operations and the importance of operational flexibility.
Good overviews. Wish it was more thorough or provided examples. Is there a list of
resources or contractors/consultants that could be provided for each state and what
services they provide?
I really can't think of anything, this was a very comprehensive manual.
Include checklists, one-page "at-a-glance" tables, etc.
The more specific and less vague it is, the more valuable a resource it will be.
No suggestions at this time.
Nothing
12. Please rate your overall satisfaction with the Manual as a compliance assistance tool.
Answer Options
Very Dissatisfied
Dissatisfied
Neither Dissatisfied nor Satisfied
Satisfied
Very Satisfied
No Opinion
Response
Percent
0.0%
0.0%
14.3%
76.2%
9.5%
0.0%
answered question
skipped question
Response
Count
0
0
3
16
2
0
21
3
November 2009
Page 14
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE
13. If EPA were to update the Manual, which update frequency would be the most useful?
Answer Options
Annually
Every 3 years
Every five years
Other timeframe (please specify)
I don't intend to refer to the Manual again
Other (please specify)
When regulations change
Any time there has been a major regulatory change
As often as changes are made
answered question
skipped question
Response
Percent
28.6%
47.6%
9.5%
14.3%
0.0%
Response
Count
6
10
2
3
0
21
3
14. What other tools and approaches would you prefer for sharing compliance assistance
information concerning ethanol facilities?
Yearly EPA meetings with industry - compliance/update seminars
Think it has been good so far.
Workshops/meetings with open, unbiased, and frank discussions between regulators
and ethanol plants.
Email listserv for ethanol industry contacts that sends regulatory/policy changes
applicable to the industry or NAICS code. Send industry compliance advisories that give
helpful info on what are some compliance concerns along with any compliance
assistance info or best practices.
An associated question and answer document to is mainted so you could access it to see
if there are any changes to the manual.
EPA inspection without fines (voluntary)
Email updates/newsletters
An email newsletter with reminders, updates and new information would be helpful.
new regulations and laws that come up every year
No suggestions. Website and manual are both sufficient in my opinion.
VI. FINAL QUESTIONS!
15. What category best describes your position?
Answer Options
Facility manager
Facility compliance officer
Other facility role (please specify)
Contractor
Response
Percent
19.0%
61.9%
14.3%
4.8%
Response
Count
4
13
3
1
November 2009
Page 15
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
If other:
EH&S Manager (2)
Consultant
Environmental engineer/EHS manager
Environmental, Health and Safety Coordinator
answered question
skipped question
21
3
16. Can we identify your facility or company (by name) as having participated in this survey?
Answer Options
Yes
No
Response
Percent
52.2%
^^ 47.8%
answered question
skipped question
Response
Count
12
11
23
1
Optional Section: if you would like to provide more specific feedback on each Section of the
Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer
the following questions.
1. Please rank your overall satisfaction with the ease of understanding, information thoroughness,
and overall usability of each of the Manual's sections:
Answer Options
Section I.I:
Requirement for
Renewable Fuels
Section 1.2:
Renewable Fuel
Standard Program
Section 1.3: Brief
Overview of Ethanol
Production Process
Answer Options
Section 1.1: National
Environmental Policy
Act
Section 1.2: Clean
Dissatisfied
0(0%)
0 (0%)
Neither
Dissatisfied «-£"- Satisned ^sfTed Option
(28.6%) 1 (14-3%) (42.9%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
2 (28.6%)
Satisfied
1 (14.3%)
1 (14.3%)
2 (28.6%)
°(0%)
(28.6%)
Neither
Dissatisfied «-??« Satisfied JSL Op^on
0 (0%)
questic
skipped question
Very
(28.6%)
answered question 7
17
0(0%)
0 (0%)
Satisfied
0(0%)
1 (14.3%)
(71.4%) °(0%) (28.6%)
November 2009
Page 16
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Air Act
Section 1.3: Clean
Water Act
Section 1.4: Safe
Drinking Water Act
Section 1.5:
Pollution Prevention
Act
Answer Options
Section 2.1: Clean Air
Act
Section 2.2:
Emergency Planning
and Community Right
to Know Act
Section 2.3: Clean
Water Act
Section 2.4: Safe
Drinking Water Act
Section 2.5: Resource
Conservation and
Recovery Act -
Hazardous Waste
Answer Options
Appendix A (First
half): Summary of
Laws Pertaining to
Ethanol Production
and Contacts
Appendix A (Second
half): Contact
Information
Appendix B:
Emergency Planning
Appendix C: Clean Air
Act Section 112-r
requirements
Appendix D:
Definitions of
0 (0%)
0 (0%)
0 (0%)
Very
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
Very
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
Dissatisfied
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
Dissatisfied
1 (14.3%)
0 (0%)
1 (14.3%)
0 (0%)
0 (0%)
1 (14.3%)
3 (42.9%)
1 (14.3%)
Neither
Dissatisfied
nor
Satisfied
1 (14.3%)
1 (14.3%)
1 (14.3%)
1 (14.3%)
1 (14.3%)
Neither
Dissatisfied
nor
Satisfied
1 (14.3%)
2 (25.0%)
2 (28.6%)
3 (42.9%)
1 (14.3%)
(42.9%) (14.3%) (28.6%)
222
(28.6%) (28.6%) (28.6%)
(28.6%) ° (0%) (28.6%)
4 2
(57.1%) °(0/o) (28.6%)
answered question 7
skipped question 17
Satisfied Satisfied Opinion
6
fQC -JO/ \ 0(0 /o) 0(0/0)
(O3./ /O)
5
(85.7%)
(71.4%) (14.3%) 0(°°/o)
(71.4%) ° (0%) (14.3%)
(71.4%) (14.3%) °(°0/0)
answered question 7
skipped question 17
-_,.. f. . Very No
satisfied Satisfied opinion
2(28.6%) (4230/o) 0(0%)
2(25.0%) (5040%) 0(0%)
4 (57.1%) 0 (0%) 0 (0%)
4 (57.1%) 0 (0%) 0 (0%)
5(71.4%) M * 0(0%)
November 2009
Page 17
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual
Report Appendices
Acronyms
Appendix E: The NEPA
Process
0(0%) 0(0%) 4(57.1%) 3(42.9%) 0(0%) 0(0%)
answered question 8
skipped question 16
Do you have any specific suggestions for improving the usefulness of any of the Manual sections?
(optional)
Answer Options
Yes
No
If yes, please provide:
Response
Percent
0.0%
100.0%
answered question
skipped question
Response
Count
0
8
0
8
16
November 2009
Page 18
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Appendix C. Interview Guide for Ethanol Facilities
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to
collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to
ethanol facilities. For the interview, it is not expected that you will have carefully read the entire Manual,
but please at a minimum review those parts of the manual that pertain to your area of work. The manual
can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf
If you have an questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone
at 206-447-1805 or by email at anna.williams@ross-assoc.com or amy.wheeless@ross-assoc.com.
TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL
Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?
1. Where did you obtain the Manual?
2. How could distribution of or access to the Manual be improved?
TOPIC AREA II: EFFECTIVENESS OF THE MANUAL
Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?
1. How readable and comprehendible is the Manual?
2. How comprehensive is the Manual in its scope?
3. What was most/least useful or informative to you in the Manual?
4. Is the contact information provided in the Appendices helpful? How could it be improved?
5. How has your facility used the Manual? If your facility has not used the Manual, why not?
6. What regulatory actions did your facility take, or intend to take, as a result of reviewing the
Manual?
November 2009 Page 19
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
7. What process or management changes did your facility make or intend to make as a result of
reviewing the Manual?
8. Did your facility make any changes in constructing or establishing the plant as a result of
reviewing the Manual?
TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE
Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?
1. After reviewing the Manual, do you feel more aware of the applicable Federal regulations?
2. Are there still federal laws or regulations that you do not understand after reading the Manual?
3. Have there been any changes in your compliance results since reviewing the Manual?
4. What "beyond compliance" measures has your facility made as a result of using the manual?
5. Are there specific environmental health outcomes (e.g. amount of waste generated) that have
resulted from use of the Manual?
6. Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that
have resulted from this Manual, or are you aware of any potential effects on human health?
TOPIC AREA IV: IMPROVING THE MANUAL
Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?
1. What changes would you recommend to the Manual to improve its readability and navigation?
2. What could be changed about the Manual to make it more thorough and comprehensive and to
make it more effective as a compliance tool?
3. Are there topics you would like more or new information about in a future Manual?
4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
or make facilities and other stakeholders aware that an updated Manual is available?
5. Do you have any other suggestions for improving the Manual?
TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE
Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?
November 2009 Page 20
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
1. What are other ways you find information about regulations and applicable laws?
2. What are other ways EPA could present the information in the Manual to its targeted audiences?
(e.g. workshops, interactive Web site)
FINAL QUESTION
1. Do you have any final "closing thoughts" or recommendations regarding the overarching
questions, sub questions or the compliance assistance Manual in general?
Thank you for participating!
November 2009 Page 21
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Appendix D. Interview Guide for Ethanol Contractors
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf
If you have an questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.
TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL
Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?
1. Where did you obtain the Manual?
2. How could distribution of or access to the Manual be improved?
TOPIC AREA II: EFFECTIVENESS OF THE MANUAL
Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?
1. How readable and comprehendible is the Manual?
2. How comprehensive is the Manual in its scope?
3. What was most/least useful or informative to you in the Manual?
4. Is the contact information provided in the Appendices helpful? How could it be improved?
TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE
Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?
1. After reviewing the Manual, do you feel more aware of the applicable Federal regulations?
November 2009 Page 22
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
2. Are there still federal laws or regulations that you do not understand after reading the Manual?
3. Are there specific environmental health outcomes (e.g. amount of waste generated) that have
resulted from use of the Manual?
4. Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that
have resulted from this Manual, or are you aware of any potential effects on human health?
TOPIC AREA IV: IMPROVING THE MANUAL
Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?
1. What changes would you recommend to the Manual to improve its readability and navigation?
2. What could be changed about the Manual to make it more thorough and comprehensive and to
make it more effective as a compliance tool?
3. Are there topics you would like more or new information about in a future Manual?
4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
or make facilities and other stakeholders aware that an updated Manual is available?
5. Do you have any other suggestions for improving the Manual?
TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE
Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?
1. What are other ways you find information about regulations and applicable laws?
2. What are other ways EPA could present the information in the Manual to its targeted audiences?
(e.g. workshops, interactive Web site)
FINAL QUESTION
1. Do you have any final "closing thoughts" or recommendations regarding the overarching
questions, sub questions or the compliance assistance Manual in general?
Thank you for participating!
November 2009 Page 23
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Appendix E. Interview Guide for State Compliance Staff
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf
If you have any questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.
TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL
Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?
1. Where did you obtain the Manual?
2. Does your agency refer ethanol facilities and contractors to the Manual?
3. How could distribution of or access to the Manual be improved?
TOPIC AREA II: EFFECTIVENESS OF THE MANUAL
Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?
1. How readable and comprehendible is the Manual?
2. How comprehensive is the Manual in its scope?
3. Is the contact information provided in the Appendices helpful? How could it be improved?
4. Have you been contacted by an ethanol facility or contractor as a result of the Manual?
TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE
Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?
November 2009 Page 24
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
1. Could compliance with state laws and rules be enhanced using the Manual? How?
2. In your interactions, do facilities seem aware of the applicable Federal regulations?
3. Have you noticed any changes in facility compliance with regulations since the Manual was
released?
TOPIC AREA IV: IMPROVING THE MANUAL
Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?
1. What changes would you recommend to the Manual to improve its readability and navigation?
2. What could be changed about the Manual to make it more thorough and comprehensive and to
make it more effective as a compliance tool?
3. Are there topics you would like more or new information about in a future Manual?
4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
or make facilities and other stakeholders aware that an updated Manual is available?
5. Do you have any other suggestions for improving the Manual?
TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE
Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?
1. What are other ways EPA could present the information in the Manual to its targeted audiences?
(e.g. workshops, interactive Web site)
FINAL QUESTION
1. Do you have any final "closing thoughts" or recommendations regarding the overarching
questions, sub questions or the compliance assistance Manual in general?
Thank you for participating!
November 2009 Page 25
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
Appendix F. Interview Guide for EPA Staff
A NOTE TO INTERVIEWEES
A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The
interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws
Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect
feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol
facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at
a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at:
http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf
If you have any questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-
447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com.
TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL
Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the
Manual to ethanol facilities and other intended audiences?
1. Does your agency refer ethanol facilities and contractors to the Manual?
2. How could distribution of or access to the Manual be improved?
TOPIC AREA II: EFFECTIVENESS OF THE MANUAL
Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility
technical support) information to ethanol facilities?
1. How readable and comprehendible is the Manual?
2. How comprehensive is the Manual in its scope?
3. Is the contact information provided in the Appendices helpful? How could it be improved?
4. Have you been contacted by an ethanol facility or contractor as a result of the Manual?
TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE
Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior
change to support improved compliance?
1. In your interactions, do facilities seem aware of the applicable Federal regulations?
November 2009 Page 26
-------
Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices
2. Have you noticed any changes in facility compliance with regulations since the Manual was
released?
TOPIC AREA IV: IMPROVING THE MANUAL
Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol
facilities?
1. What changes would you recommend to the Manual to improve its readability and navigation?
2. What could be changed about the Manual to make it more thorough and comprehensive and to
make it more effective as a compliance tool?
3. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences,
or make facilities and other stakeholders aware that an updated Manual is available?
4. Do you have any other suggestions for improving the Manual?
TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE
Overarching Question: What approaches other than the Manual would provide effective compliance assistance
to the ethanol facilities?
1. What are other ways EPA could present the information in the Manual to its targeted audiences?
(e.g. workshops, interactive Web site)
FINAL QUESTION
1. Do you have any final "closing thoughts" or recommendations regarding the overarching
questions, sub questions or the compliance assistance Manual in general?
Thank you for participating!
November 2009 Page 27
------- |