United States Office of Policy, Environmental Protection Economics and Innovation Agency (1807T) November 2009 EPA-100-R-10-001 Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Promoting Environmental Results 4 Through Evaluation ------- Acknowledgements This evaluation was performed by Ross & Associates Environmental Consulting, Ltd. (Ross & Associates) under contract to Industrial Economics, Incorporated (lEc) for EPA's Office of Policy, Economics and Innovation (OPEI) under Contract EP-W-07-028 between EPA and I EC. The evaluation team included Anna Williams, Tamara Power-Drutis and Amy Wheeless of Ross & Associates, Ashley Betts and Ward Burns of EPA Region 7 and Michelle Mandolia of OPEI's Evaluation Support Division. This report was developed under the Program Evaluation Competition, sponsored by OPEI. To access copies of this or other EPA program evaluations, please go to EPA's Evaluation Support Division's website at http:www.epa.gov/evaluate. ------- Table of Contents Executive Summary i I. Introduction 1 Background 1 Purpose of the Evaluation 2 Report Organization 2 II. Methodology 3 Evaluation Approach 3 Information Collection and Analysis 4 III. Findings 8 Topic Area I: Outreach and Access 8 Topic Area II: Effectiveness of the Manual as an Information Tool 11 Topic Area III: Changes in Understanding and Behavior 14 Topic Area IV: Improving the Manual 16 Topic Area V: Other Compliance Assistance 19 IV. Recommendations 20 Appendices (Available in Separate Document) A. Customer Satisfaction Survey B. Customer Satisfaction Survey Results C. Interview Guide - Ethanol Facility D. Interview Guide-Contractor E. Interview Guide - State Compliance Staff F. Interview Guide - EPA Staff ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Executive Summary Background and Evaluation Process Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were issued new environmental permits. The increase in new ethanol facilities was especially significant given that, during the previous 20 years, only 30 ethanol plants had been permitted. Region 7 recognized that the ethanol production sector may need targeted outreach and education on the pertinent federal environmental rules, regulations, and programs that govern the sector's construction and operation. To meet this need, in November 2007, Region 7 staff published a compliance assistance manual for new ethanol facilities entitled Environmental Laws Applicable to Construction and Operation of Ethanol Plants (hereto after referred to as "the Manual"). The Manual included information on federal environmental programs and the roles of state and federal authorities in overseeing ethanol facility construction and operation. The Manual was primarily targeted to ethanol plants, but was also likely of interest to state regulatory program offices, trade associations, consultants, and the public. Region 7 conducted a number of outreach activities to publicize the Manual to ethanol plants and other stakeholders. Region 7, with support from the EPA Office of Policy, Economics and Innovation (OPEI), wished to evaluate the success of the Manual in improving industry compliance with relevant rules and regulations and to gather information on the Manual's readability, quality of information, and overall usefulness as a compliance assistance tool to ethanol facilities in the Region. In addition, Region 7 wanted to identify ways to improve facility satisfaction with the Manual as a compliance assistance tool and to determine what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of this evaluation are intended primarily for Region 7 staff as they prepare revisions to the Manual and produce other outreach materials for the ethanol production sector. An EPA-contractor team led the evaluation. Two representatives from Region 7 were part of the team; one who had helped to develop the Manual as a member of the Region 7 Biofuels Team, and one representative of the Region's policy and management division. The other EPA team member was an evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI. The contractor team Exhibit ES-i. Overarching Evaluation Questions Topic Area I: Outreach and Access. How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? Topic Area II: Effectiveness of the Manual as an Information Tool. How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? Topic Area III: Changes in Understanding and Behavior. Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? Topic Area IV: Improving the Manual. How could the Manual be improved to provide effective compliance assistance to ethanol facilities? Topic Area V: Other Compliance Assistance. What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? November 2009 ES-i ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report consisted of Industrial Economics (lEc) and a subcontractor, Ross & Associates Environmental Consulting, Ltd. (Ross & Associates). Ross & Associates implemented most of the contractor responsibilities and produced this report with support from the other team members. The evaluation focused on five overarching evaluation questions, listed on the previous page in Exhibit ES-1. To answer the overarching questions, the evaluators conducted 20 interviews with facility environmental managers, ethanol industry contractors, state compliance staff members, and EPA staff members. In addition, the evaluators developed a customer satisfaction survey and distributed the survey to ethanol facilities in Region 7 and ethanol industry contractors known to work in the Region. Finally, the evaluators conducted research on ethanol plant compliance to try to answer the overarching evaluation question on the effect the Manual had on facility compliance. Findings Those interviewed and surveyed for the evaluation found the Manual to be very useful. They appreciated its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the proactive assistance approach embodied in the Manual represented. Generally, comments for improvement focused on timelier and improved distribution and a variety of changes to make the Manual a stronger compliance assistance tool. Below were the major findings for each overarching evaluation topic area. Topic Area I: Outreach and Access Awareness of the Manual amongst the primary audiencesethanol facilities and contractorswas not as prevalent as it could be despite outreach efforts by Region 7 staff. In addition, the timing of the Manual's publication occurred after most new plants had commenced or completed construction; only a handful of new ethanol plants started productions since November 2007. A majority of the facilities that the evaluators spoke with in the summer of 2009 were not immediately aware of the Manual's existence. Nonetheless, most of these facilities were receptive to reviewing the Manual as a part of this evaluation. Feedback also indicated that the audience for the Manual is larger than just Region 7-specific ethanol facilities and contractors, and that promoting the Manual nationally could be beneficial, particularly to EPA Regions with significant numbers of ethanol plants. Topic Area II: Effectiveness of the Manual as an Information Tool Participants in the evaluation found the Manual to be well organized, easy to navigate, and an appropriate tool for conveying compliance information to facilities. Having this material, including appropriate contacts, organized in one place provided for an easy reference for facility managers and they found the Manual to be readable and organized logically. Participants provided varied and inconsistent feedback on whether the Manual was too technical or not technical enough, but everyone interviewed and surveyed indicated that they found the Manual to be useful in some way. Whether the Manual has the appropriate level of detail likely depends on the reader's existing level of knowledge and facility needs. Topic Area III: Changes in Understanding and Behavior All of the facilities and contractors that participated in this evaluation said the Manual was comprehensive and was helpful for increasing or solidifying their understanding of environmental compliance requirements. This evaluation cannot conclusively say that the Manual has led to any significant changes that would affect facility compliance. With only a few exceptions, respondents to this evaluation were not able to identify specific instances where facilities had more awareness of applicable federal regulations or had made compliance or management changes as a result of the Manual. Research of the EPA Enforcement November 2009 ES-ii ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report and Compliance History Online (ECHO) database showed that, after the Manual was published in late 2007, there was some change in informal and formal enforcement actions at Region 7 ethanol facilities, but due to lagging and incomplete enforcement data and the inability to know exactly when a facility reviewed the Manual, the evaluators cannot say conclusively than the Manual had an effect on compliance. Topic Area IV: Improving the Manual Respondents to this evaluation had a number of suggestions on how future versions of the Manual could be improved. These suggestions centered around additional content, such as a section on cellulosic ethanol, or around functional content, such as checklists, case studies, and specific examples. The recommendations below carry forward many of the suggestions offered by those interviewed and surveyed. Topic Area V: Other Compliance Assistance Evaluation participants thought that, while the Manual itself was a useful tool, other venues, such as conferences, EPA workshops, a more interactive Web site, and one-on-one interaction with ethanol facilities were also valuable approaches for providing compliance assistance to ethanol facilities. Recommendations Evaluation participants clearly conveyed that the Manual was useful and valuable and that it effectively communicated relevant information for ethanol plants. The following recommendations focus on ways to improve the Manual for any potential future versions and on other ways that Region 7 could make the existing information available to its intended audiences. The evaluators understand that there may be limited resources available for updating the Manual and limited time available for staff to provide other kinds of compliance assistance to ethanol facilities. Further, the market for ethanol has been shifting dramatically and demand for new facility permits has not returned to the peak it was in when the Manual was first released. Demand for the Manual and feasibility of implementing any of the evaluation's recommendations will likely be influenced by these and other factors. The following recommendations, which are expanded upon in the body of the report, are offered for Region 7's consideration. 1. As a first step, consider whether to tailor the Manual's content and distribution to the national level. 2. Consider additional outreach strategies and targeted audiences. 3. Consider making this tool primarily Web-based, with a hard copy Manual as a supplement. 4. Update the Manual to include new regulatory and compliance information. 5. Create an executive summary to encapsulate the main points of the Manual. 6. Incorporate more "at a glance" information, such as lists of resources, call-out boxes, checklists, and calendars. 7. Provide more specific examples and case studies. 8. Expand content to include regulatory information about topics of interest to ethanol plants, such as cellulosic ethanol production and byproduct diversion into other products. 9. Expand information on beyond compliance efforts and best practices. 10. Apply the lessons from this evaluation to the Region's other compliance assistance and beyond compliance efforts. November 2009 ES-iii ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report I. Introduction Background Between June 2005 and June 2008, approximately 100 ethanol plants in EPA Region 7 (Region 7) were issued new environmental permits. The increase in new ethanol facilities was especially significant given that, during the previous 20 years, only 30 ethanol plants had been permitted. Region 7 recognized that the ethanol production sector may need targeted outreach and education to understand the pertinent federal environmental rules, regulations, and programs that govern the sector's construction and operation. To meet this need, in November 2007, Region 7 staff published a compliance assistance manual for new ethanol facilities entitled "The Manual in general was Environmental Laws Applicable to Construction and Operation of a huge undertaking and a Ethanol Plants (hereto after referred to as "the Manual"). The Manual great idea... a good public included information on federal environmental programs and the roles | service..." - EPA contact of state and federal authorities in overseeing ethanol facility construction and operation.1 Ethanol plants were the primary audience for the Manual, but the Manual was also likely of interest to state regulatory program offices, trade associations, consultants and the public. To distribute the Manual to the relevant stakeholders, Region 7 staff conducted a number of outreach activities: Staff announced the development of the Manual at the 2007 National Association of Farm Broadcasting Convention and distributed ethanol materials to attendees and exhibitors. Staff developed a news release about the development of the Manual and distributed it to those who had expressed interest and to the editors of agriculture and biofuels magazines. Staff posted the Manual on the EPA Region 7 Web site. The Region 7 Office of Public Affairs (OPA) shared ethanol postcards with the Chairs of the Missouri House Agriculture Committee, the Agribusiness Committees, and the Senate Agriculture Committee, along with members of these committees, as well as at relevant conferences (e.g., Commodity Class, the Missouri Agribusiness Association annual meeting, the Kansas Feed and Grain annual conference, the Nebraska Governor's Conference on Agriculture, and the Missouri Governor's Conference on Agriculture). The Region 7 Regional Administrator (RA) and OPA attended the Fuel Ethanol Workshop and announced the development of the Manual. OPA collected contact information for those interested in the Manual and notified them via email when Region 7 released the Manual. The RA conducted news interviews about the Manual (e.g., Missourinet, WHO-Farm radio) and mentioned the Manual in a number of speeches (e.g., at Commodity Classic, Bunge North America, Inc., CenSARA's National Environmental Biofuels Conference). OPA met with executives of ethanol plants and shared information about the Manual. 1 A copy of the Manual can be downloaded at: http://www.epa.qov/reqion07/priorities/aqriculture/ethanol plants manual.pdf November 2009 Page 1 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Air program staff discussed the Manual at a number of events and meetings (e.g., Iowa State Field Briefing, a meeting with the head of EPA's Office of Enforcement and Compliance Assistance, CenSARA Director's Meeting). Staff informed a number of interested stakeholders about the Manual, including RCRA (Resource Conservation and Recovery Act) State programs, the U.S. Army Corps of Engineers. Staff also informed two ethanol facilities about the Manual during the facilities' RCRA inspections. Each Emergency Planning & Community Right-to-Know Act (EPCRA) state coordinator announced the availability of the Manual at his or her State Emergency Response Commission meetings. Purpose of the Evaluation Region 7, with support from the EPA Office of Policy, Economics, and Innovation (OPEI), wished to evaluate the success of the Manual in improving industry compliance with relevant rules and regulations and to gather information on the Manual's readability, quality of information, and overall usefulness as a compliance assistance tool to ethanol facilities in the Region. In addition, Region 7 wanted to identify ways to improve facility satisfaction with the Manual as a compliance assistance tool and to determine what other compliance assistance tools and materials may be helpful to ethanol facilities. The results of this evaluation are intended primarily for Region 7 staff as they prepare revisions to the Manual and produce other outreach materials for the ethanol production sector. An EPA-contractor team led the evaluation. Two representatives from Region 7 were part of the team; one who had helped to develop the Manual as a member of the Region 7 Biofuels Team, and one representative of the Region's policy and management division. The other EPA team member was an evaluation specialist in EPA Headquarters' Evaluation Support Division within OPEI. The contractor team consisted of Industrial Economics (lEc) and a subcontractor, Ross & Associates Environmental Consulting, Ltd. (Ross & Associates). Ross & Associates implemented most of the contractor responsibilities and produced this report with support from the other team members. The results of this evaluation are intended primarily for Region 7 staff as they prepare revisions to the Manual and produce other outreach material for the ethanol production sector. The results of this evaluation could also assist this and other Regions with compliance assistance materials for industry and manufacturing sectors. Report Organization This report consists of four sections, in addition to this Introduction (Section I). Section II provides an overview of the evaluation methodology. Section III describes the findings for each evaluation question. Section IV lists and describes the evaluation's concluding recommendations for Region 7. Appendices, which include a copy of the survey, the survey results, and the interview guides, are provided in a separate document. November 2009 Page 2 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report II. Methodology The following section provides an overview of the evaluation methodology, including the team's initial approach for planning the evaluation and information collection and analysis process. Evaluation Approach As a first step in the evaluation, the evaluation team refined the initial evaluation questions that had been submitted as a part of OPEI's evaluation competition, identified the information needs to answer the refined questions, and updated the draft ethanol manual logic model that had been developed previously to help inform the evaluation. Evaluation Questions The evaluation team developed the overarching evaluation questions listed below to structure the evaluation. The overarching questions cover five topics: effectiveness of distribution of to the Manual, effectiveness of the Manual as an information tool, effectiveness of the Manual on understanding and behavior changes, possible improvements to the Manual, and other approaches for providing compliance tools. Exhibit i. Overarching Evaluation Questions Topic Area I: Outreach and Access. How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? Topic Area II: Effectiveness of the Manual as an Information Tool. How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? Topic Area III: Changes in Understanding and Behavior. Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? Topic Area IV: Improving the Manual. How could the Manual be improved to provide effective compliance assistance to ethanol facilities? Topic Area V: Other Compliance Assistance. What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? These overarching evaluation questions informed data collection, including development of the interview guides and the survey (see Appendices). This report presents findings according to each overarching evaluation question. Evaluability Assessment Following development of the evaluation questions, the evaluation team conducted an evaluability assessment to ensure that there would be enough information to assess the effectiveness of the Manual. This evaluability assessment identified: What information would be needed to address the key evaluation questions; Whether the needed information would be available; and How the available information would be collected. November 2009 Page 3 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report The conclusion of this evaluability assessment was that, with the available data and information collection methods, to some extent all evaluation questions would be answerable, with qualitative, yet authoritative, information and stakeholder feedback. The one exception was Topic Area III: the evaluability assessment concluded that, due to a lack of data, the evaluation would not be able to draw definitive conclusions about a causal relationship between the Manual and facility compliance. Logic Model In preparing for this evaluation, Region 7 developed a logic model for the Manual and its associated activities. The evaluation team refined the model to elaborate on potential outcomes of Manual use (see Exhibit 2 on the following page). The final logic model depicts the linkages between program activities inputs, outputs, stakeholders, and expected outcomes of Region 7's outreach activities and compliance assistance tool. Information Collection and Analysis In planning for the evaluation, the evaluation team determined that interviews with relevant stakeholders, supplemented by a survey of facilities and contractors, would help to answer the overarching evaluation questions. The primary target audience for the Manual and thus the main source of feedback were ethanol facilities and contractors. The secondary audiences were state and EPA Region 7 compliance staff. While trade associations and the public were also listed in the logic model as an "audience" or "customers" for the Manual, they were not targeted for separate information collection in this evaluation. Using information provided by Region 7, the evaluators contacted the ethanol plants and a handful of contractors to assess interest in participating in an interview and/or a survey. During this contact, the evaluators also offered to send Internet links for the Manual to those facilities and contractors who were not previously aware of the Manual or had not received it. In addition to interviews and surveys, the evaluators conducted some limited research on facility compliance in the Region to better understand any changes in facility compliance. Interview Approach The evaluators collected data from ethanol plants, contractors, EPA staff, and state compliance staff through a set of telephone interviews. Facility and contractor interviews were with individuals who confirmed that they had received and reviewed the Manual. Region 7 staff contributed suggestions on who to interview at EPA and state agencies. The list of interviewees was reviewed by other members of the evaluation team to ensure a representative selection. The final set of interviewees consisted of: Individuals at seven facilities that had responsibilities pertaining to environment compliance. Three ethanol-industry environmental consultants/contractors; an additional consultant submitted written feedback. Five state compliance staff members from three Region 7 state environmental agencies. Five EPA staff members, four of whom were from Region 7 and one from EPA Headquarters. November 2009 Page 4 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Exhibit 2. Ethanol Compliance Manual Logic Model Goal: Improve environmental regulatory compliance among ethanol facilities in order to reduce impacts on environmental and human health Resources from: EPA Region 7 Biofuels Team EPA Region 7 Ag Team i State Agencies Trade Groups ix ACTIVITIES Create Compliance Manual Answer Regulatory Questions Develop Other Outreach Materials Improve and Revise Manual PARTICIPANTS Ethanol Facilities Biofuels Team and Regulatory Programs State Compliance Staff Ethanol Contractors Trade Groups i ix SHORT TERM Improved Awareness and Knowledge of Regulatory Requirements and Attitude Change OUTCOMES MEDIUM TERM Changes in Ethanol Facilities Operations or Management LONG TERM2 Improved Facility Compliance Reduced Environmental and Human Health Impacts OUTCOME MEASURES # of facilities that say they better understand how to comply with regulations # of facilities that contacted someone for further compliance assistance # of Manuals accessed (print copies/ downloads) # of facilities that took at least one recommended action to comply with regulations # of facilities that adopted process changes # of facilities improving environmental management systems or conducting reviews Compliance rate changes #of facilities that have changed regulatory status #of facilities that eliminated, treated or reduced emissions or other pollutants Quantified environmental improvements It was assumed that increased compliance with environmental laws would result in the long-term outcomes of improved environmental and public health and that these long-term outcomes would be realized over the course of months or years. It was beyond the scope or ability of this evaluation to identify whether these outcomes have occurred as a direct result of the compliance assistance manual in question. Similarly, it was not feasible as part of this evaluation to gather the data needed to develop the long-term (aspirational) performance measures listed in the logic model. November 2009 Page 5 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Interviewees received interview guides (see Appendices C, D, E, and F) in advance of their interviews. Interviews were generally conducted by one Ross & Associates staff member and lasted 15 to 30 minutes. The evaluators conducted the telephone interviews based on the interview guides; however, the order in which topics were discussed sometimes varied. Notes from these interviews were collected during the interviews and entered in an Excel workbook. These notes were considered confidential and not included in this report. Any specific quotes that this report uses as examples were sent to the relevant interviewees to ensure accurate communication of ideas, and are not attributed to the individual in this report. The evaluation team had originally considered conducting a set of follow-up interviews with the primary and secondary audiences after doing an initial round of information collection. However, after conducting the interviews, the evaluation team determined that follow-up interviews would be unlikely to yield additional feedback or insight. However, the evaluation team did meet with members of the EPA Region 7 Biofuels Team to discuss the report's draft findings and recommendations. Survey Approach There were as many as 150 ethanol facilities in Region 7, as well as a number of ethanol industry contractors, who may have received and read the Manual. To reach as many of them as possible and understand their access to and satisfaction with the Manual, the evaluation team developed and administered a written customer satisfaction survey to be completed by mail or on the Internet. The Federal Paperwork Reduction Act of 1995 requires that federal agencies receive approval from the Office of Management and Budget (OMB) before requesting information from more than nine non-federal entities, and must do so by submitting an Information Collection Request (ICR) to OMB for approval. The evaluation team developed a survey, submitted the survey under generic ICR 1711.05 to OMB, and received approval following some minor survey revisions. Following OMB approval of the survey, Ross & Associates emailed or mailed the survey to all Region 7 ethanol facilities and contractors for which it had contact information. This survey (see Appendix A) focused on customer satisfaction with the Manual. Survey responses were anonymous unless respondents chose to identify themselves. Participants were encouraged to respond to the survey within approximately four weeks. Approximately three weeks after the survey was mailed, the evaluators contacted facilities and contractors with a reminder email or reminder postcard and extended the due date to solicit more responses. The evaluators mailed or emailed surveys to approximately 120 facilities and contractors. Of these, approximately ten were returned as having no recipient. Of the remaining, 24 facilities and contractors completed surveys; at least three had also participated in a telephone interview. Others may have participated in telephone interviews, but did not identify themselves in the survey. The response rate for the survey was approximately 20 percent; generally, response rates for customer satisfaction surveys range from 10 percent to 30 percent, so this response rate was not unexpected. Factors that may have contributed to this response rate include the fact that the evaluators did not connect with all facilities and contractors to alert a specific contact that a survey would be forthcoming; some facilities in Region 7 were no longer in operation; and there was no tangible and immediate incentive offered for completing the survey. Additional Information Review In addition to the surveys and interviews, the evaluators reviewed background material and researched compliance information through available data to help answer the overarching evaluation questions. The evaluators reviewed the Manual itself, the outreach activities that Region 7 had done to promote November 2009 Page 6 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report distribution of the Manual, and the plant list available at Ethanol Producer Magazine (http://ethanolproducer.com/plant-list.isp'), which provides status of operation, when the facility started production, and type of feedstock. In addition, the evaluators conducted Internet searches to see which organizations or companies had posted links to the Manual from their Web sites. Finally, the evaluators reviewed EPA's Enforcement and Compliance History Online (ECHO) database. ECHO provides an overview of a facility's environmental record under the Clean Air Act (CAA) Stationary Source Program, the Clean Water Act (CWA) National Pollutant Elimination Discharge System (NPDES), and the Resource Conservation and Recovery Act (RCRA), and associated state and federal responses. The evaluators queried each facility in Region 7 that had information in the database and tracked informal and formal violations pre-publication of the Manual and post-publication of the Manual (the Manual was published in November 2007). The evaluators also looked at the compliance history of specific facilities that were interviewed or surveyed and compared the results for a richer analysis of compliance. November 2009 Page 7 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Findings The following pages describe the evaluation's key findings, organized by the overarching evaluation questions. Those interviewed and surveyed for the evaluation found the Manual to be very useful. They appreciated its existence, the effort that EPA Region 7 undertook to develop it, and the "gesture" that the proactive assistance approach embodied in the Manual represented. Generally, comments for improvement focused on timelier and improved distribution and a variety of changes to make the Manual a stronger compliance assistance tool. Topic Area I: Outreach and Access How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? Overarching Findings As discussed in Section I, Region 7 staff conducted a number of outreach activities to publicize the availability of the Manual to ethanol plants and other interested stakeholders. However, awareness of the Manual was not as prevalent as it could be for these audiences, and the timing of its publication was not coincident with the largest increase in construction of ethanol plants. Nonetheless, access to the Manual was sufficient for stakeholders who were aware of it, and many had additional suggestions for how to improve advertisement of the Manual. Feedback also indicated that the audience for the Manual was larger than just Region 7-specific ethanol facilities and contractors, and that promoting the Manual nationally could be beneficial, particularly to EPA regions with significant numbers of ethanol plants. /Access to Manual: Overall, for facilities and contractors aware of the Manual, access to the Manual was considered sufficient, with many receiving it from an EPA or state contact or at a workshop or conference. However, a majority of facilities (31 of 51) that the evaluators spoke to in the initial evaluation contact period were not immediately aware of the Manual's existence. Nonetheless, most of these facilities were receptive to receiving a Web link to the Manual. Of these 51 facilities, eight had started production in 2008 or 2009; only one of these newer facilities indicated awareness of the Manual prior to the evaluation. Timing: Many interviewees noted that the timing of publication of the Manual was not idealfew ethanol facilities have been constructed since November 2007, after the Manual was published, and many have gone out of business or halted production since that time. Distribution Audiences: Many facilities indicated that they use contractors and consultants to find out information on current or upcoming regulations and refer to them with questions on compliance or permitting. Facilities also I "Ethanol is not unique to Region look to other ethanol-related organizations that interact regularly 7 |f a regjon comes out with with the ethanol sector for information. These groups were something that's a good noted as venues for distribution if EPA were to develop future product, then the agency should versions of the Manual. consider marketing it beyond National Relevance: Several interviewees noted that the tnat region... It should be a information in the Manual would be well-received in other EPA national manual. The federal Regions with significant numbers of ethanol plants. In addition, regulations are all the same, so research during this evaluation found that the Manual was tney should all use the same linked on the state environmental agency Web sites of states manual."-EPA contact not in Region 7, highlighting that non-Region 7 ethanol plants November 2009 Page 8 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report may have found this information to be relevant. Interview Findings Of those that participated in the interviews, six of the seven facilities and two of the three contractors were aware of the Manual prior to being contacted for the evaluation. These interviewees indicated that they had received it from: a contact at EPA Region 7; a manager; training sessions and workshops; and an environmental consultant. Generally, the timing of the publication of the Manual was more of an issue than mode of distribution in getting information out to relevant stakeholders. As few ethanol plants have been placed into operation since the Manual was published, many of the facilities the evaluators spoke with were already aware of the information, having been in operation for some period of time. Region 7 noted that more than 100 facilities have been issued environmental permits between June 2005 and June 2008. According to Ethanol Producer Magazine, only 11 facilities have been put into operation after November 2007, indicating that the largest increase in activity was prior to publication of the Manual.3 Additionally, a number of plants have gone out of business or have halted production in recent months. According to Ethanol Producer Magazine, only seven facilities in Region 7 were not operating during the summer and autumn of 2009. However, during the evaluation, the evaluators encountered disconnected lines, returned surveys, and comments from ethanol plant staff that indicated that the number of facilities that had at least temporarily halted production was higher. Interviewees from the state agencies all work in some capacity with ethanol facilities and noted that they have not referred any facilities to the Manual, but indicated that they may in the future as the ethanol industry rebounds and new facilities are constructed. Both EPA and state staff noted that the information in the Manual was helpful to operators just beginning to plan construction of a facility, but those people are difficult to identify unless they approach regulators before beginning plant construction. For improving distribution of the Manual, contractors were highlighted as an important and oft-used resource for ethanol facilities on environmental issues. Five of the seven ethanol plants that were interviewed indicated that they receive assistance on environmental compliance and permitting issues from contractors and that these contractors alert them of new and upcoming regulations. All of the contractors also stated that they help many ethanol plants in Region 7 with their environmental compliance issues. In addition to contractors and consultants, facility interviewees noted that they received information about environmental regulations and tools from many different sources, and suggested these venues as ways to get information about to this audience in the future. Region 7 distributed the Manual using some of the below approaches (e.g., making information about the Manual available at workshops, alerting relevant stakeholders), but feedback suggested that wider distribution through a number of venues, including the following, would be useful: Make Manual available at environmental workshops. Provide Manual during EPA or state on-site technical assistance or inspections. Make information available on a more interactive EPA Web site. Distribute information on EPA and external email listservs. Post Manual on state environmental agency Web sites. Send hard copies of Manual to all potentially relevant organizations or facilities, including: 3 Ethanol Producer Magazine, Plant List. Accessed at http://ethanolproducer.com/plant-list.isp on August 18, 2009. November 2009 Page 9 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report o Directly to contractors and consultants. o State regulatory agencies. o State ethanol boards. o EPA program offices. o National and state ethanol trade associations. o National and state environmental consulting trade associations. o Air resource agencies. Finally, two EPA staff members and two contractors noted, unprompted, that the information in the Manual is not just relevant to Region 7 ethanol plants, and the Manual should be distributed and advertised more nationally. Survey Findings Overall, survey respondents were satisfied with their access to the Manual and had generally heard about it either directly from EPA, from a state contact, or during a workshop or conference. Four respondents, or about 17 percent of respondents, noted that they learned of the Manual specifically when contacted with the survey or during the initial evaluation contact period. Of the other respondents, they had learned about the Manual through visiting EPA's Web site, through their state contact, at conferences or workshops, or from an EPA postcard. None had learned about it through an EPA television or radio interview, and only one due to an EPA news release (see Figure below). All but one of the respondents, or 95 percent, indicated that they were satisfied or very satisfied with their access to the Manual; the other respondent was neither satisfied nor dissatisfied. How Respondents Learned About the Manual N=23 Don't Know/ Don't Remember 4% Other Correspondence f-c-r E=A 9% EPA news relesse 4% November 2009 Page 10 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Other Findings The evaluators researched where this Manual had been referenced on the Internet. Region 7 conducted a number of outreach activities to announce the publication of the Manual, including developing a press release. News aggregation and industry-focused Web sites and journals picked up this release and developed short news stories for their readers, including Biofuels Journal4, Enviro Business and Legal Reports5, Bulk Transporter6, and ChemAlliance7. In addition, the Manual can be found on the Nebraska Ethanol Board's and Nebraska State Department of Environmental Quality's Web sites8 9, and on the Web sites of some state departments that are not in Region 7, including Kentucky10 and Minnesota.11 The Minnesota Web site noted that, while the Manual "was not directly intended for Minnesota ethanol plants, many of the same policies and regulations apply." The Manual had received 129,337 Web server requests ("hits") from its release in November 2007 through August 2009. Topic Area II: Effectiveness of the Manual as an Information Tool How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? Overarching Findings Participants in the evaluation found the Manual to be well organized, easy to navigate, and an appropriate tool for conveying compliance information to facilities. Having this material, including appropriate contacts, organized in one place provided for an easy reference for facility managers. Participants indicated that they found the Manual to be readable and organized logically. Participants provided varied and inconsistent feedback on whether the Manual was too technical or not technical enough, but everyone interviewed and surveyed indicated that they found the Manual to be useful in some way. Whether the Manual has the appropriate level of detail likely depends on the reader's existing level of knowledge and facility needs. Readable and Organized: Overall, respondents for this evaluation found the Manual to be readable and well-organized. 4 "EPA Ethanol Manual: Environmental Compliance Manual for Ethanol Plants." Biofuels Journal. May/June 08, page 44. Accessed on September 10, 2009: http://vwwv.nxtbook.com/nxtbooks/GJ/BJ_56_08/index.php 5 "EPA Region 7 Announces Rollout of Environmental Manual for Ethanol Facilities." Enviro Business and Legal Reports. November 16, 2007. Accessed on August 19, 2009: http://enviro.blr.com/news.aspx?id=83282 6 "EPA develops ethanol manual for producers." Bulk Transporter, November 19, 2007. Accessed on August 19, 2009: http://bulktransporter.com/management/tank-truck/epa_ethanol_manual/. 7 "EPA ethanol Compliance Manual Available Online." ChemAlliance, September 5, 2008. Accessed on August 19, 2009: http://www.chemalliance.org/news/news_detail.asp?StorylD=2272 8 "Reports." Nebraska Department of Environmental Quality. Accessed on August 19, 2009: http://www.deq.state.ne.us/Publica.nsf/Pages/06-192 9 "Resources." Nebraska Ethanol Board. Accessed on September 10, 2009: http://www.ne- ethanol.org/resources/resources.htm 10 "BioFuel Considerations." Kentucky Department for Energy Development and Independence. Accessed on August 19, 2009: http://www.energy.ky.gov/dre3/renewable/considerations.htm 11 "Ethanol in Minnesota - information for industry." Minnesota Pollution Control Agency. Accessed on August 19, 2009: http://www.pca.state.mn.us/energy/fuels/ethanol-facilities.html November 2009 Page 11 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Level of Detail: Among respondents, there was disagreement over whether the document was too technical or not technical enough, but everyone interviewed and surveyed indicated that they found some use from the Manual. In addition, most respondents thought the right amount of information was available in the Manual, though a few indicated that too little information was available and provided ideas for where content could be expanded or added. Reference Guide: As noted previously, almost all facilities who responded to this evaluation had started production prior to publication of this Manual, so they were familiar with the material; however, they expressed appreciation for having the information in one place to refer to when needed. Contact Information: Respondents saw the value of having specific state and EPA contacts in Appendix A of the Manual to refer to for more information, but noted that it would be more difficult to keep up to date than other information in the Manual. Interview Findings "If anything you're trying to reach a broad audience, and the more broad you make it the less technical it becomes... so it doesn't necessarily have the right level of detail for the finer points in regulation for all intended audiences." - EPA contact All interviewees provided an overall positive assessment of the content and organization of the Manual, indicating that the Manual was organized well and was very readable. Comments included that it was helpful that the Manual was organized by EPA program areas and that the language was very straight forward. Critiques of the Manual from some respondents were that it may be slightly too technical for those new to the field of environmental compliance and permitting and that its length may prevent some potential users from perusing it. However, no interviewees indicated that they saw opportunities for shortening the Manual. Other interviewees, however, indicated the oppositethat the Manual was too general and needed more detail to be more useful to ethanol plants. One state interviewee noted that when a facility first contacts them about an environmental permit, their state agency has a multi-media group that provides information on the requirements for the facility; gives permit application forms and any other relevant handouts; and offers to have an in-person meeting for any questions. This outreach provides much of the same information found in the Manual, so the Manual itself was less useful in providing unique information to this state's facilities. All but one of the facilities interviewed had started up production prior to publication of this Manual, so the general content of the Manual was already familiar to them. The one facility that started up after the Manual was published used it as a reference guide for construction and referred to it occasionally during operation. Other respondents noted that the Manual overall was very useful, in that having all the information in one place was in itself helpful. Another facility respondent said their facility had put together a calendar based on the Manual to help track due dates for permits and plans. "There are differences between the contacts in the appendix and the contacts on the Iowa DNR Web site. In some cases, the contact section in the manual lists the head of the department, which I probably wouldn't need to contact. Before using the contact information in the manual I would check the Iowa DNR Web site to make sure I contact the appropriate person." - Ethanol facility The evaluators specifically asked all interviewees about the "Who to Contact" section of Appendix A and whether this section was useful and any ideas to improve it. Interviewees indicated that the contact information in Appendix A was helpful, though approximately half of the interviewees had not looked at this section until asked about it during the interview. A common comment on Appendix A was that it might be hard for EPA to keep the contact information current and that putting titles rather than contact names could help with this November 2009 Page 12 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report challenge. Others, however, preferred to keep the specific contact names in the Appendix and have EPA update the information frequently, perhaps on a linked Web site. One facility said that the contact information provided in the Appendix did not always match the contact information found on their state Web site or was not the appropriate level of staff person. One state interviewee said that he has received calls from around the United States with questions about ethanol regulations and wished to have a list of all state contacts for ethanol issues, rather than just those in Region 7. Survey Findings Survey respondents were generally positive about the Manual and its readability and organization. Twenty of the 21 respondents, or more than 95 percent, who answered questions on readability indicated that the Manual was somewhat easy or very easy to read (see Figure below). Satisfaction with Manual's Readability N = 21 47.6% 47.6o/0 0.0%__4. Sixteen (76 percent) of respondents said it was easy or very easy to navigate to areas of interest, while five (24 percent) said it was neither easy nor difficult to navigate the Manual. Twelve respondents (almost 60 percent) thought there was the right amount of information in the Manual (see Figure below). Amount of Information in the Manual N=21 Slightlytoo much 5% Significantly too much (0%) Significantly too little (0%) No opinion (0%) November 2009 Page 13 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Topic Area III: Changes in Understanding and Behavior Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? Overarching Findings This evaluation indicates that existence and use of the Manual has led to increased or enhanced understanding and knowledge for ethanol facilities. However, this evaluation cannot conclusively say that the Manual has led to any significant changes that would affect compliance on the part of facilities. Understanding and Awareness: All of the facilities and contractors said the Manual was comprehensive and was helpful for increasing or solidifying their understanding of environmental compliance requirements. Behavior Change: With only a few exceptions, respondents were not able to identify specific instances where facilities had more awareness of applicable federal regulations or had made compliance or management changes as a result of the Manual. Interview Findings "I used it because we were brand new, and it was great for starting up. Now that we're in production I refer back to the Manual. I also belong to a group of ethanol producers, and many times people will call me with questions. I use it as a reference tool to help answer them." - Ethanol Facility All of the facilities and contractors said the Manual was comprehensive and was at least somewhat helpful for increasing their understanding of environmental compliance requirements, or solidifying what they already knew. The contractors interviewed were all very familiar with the applicable environmental rules, but all said that they thought the Manual would be helpful to a facility starting out in the construction and operation process. State interviewees noted that if the ethanol industry rebounds, the Manual would be very useful for helping new operators understand requirements. EPA and state compliance staff were asked whether facilities seemed more aware of applicable federal environmental requirements as a result of the Manual. One EPA interviewee noted that there had been increased awareness of the Risk Management Program (Clean Air Act, Section 112 (r)); this interviewee said that, prior to the Manual, many plants did not realize they were covered under this regulation. Other compliance staff noted that there may be increased awareness among the regulated community, but that they could not conclusively point to the Manual as the cause of that knowledge. "It allowed us to not ship out work to consultants; we did all the work in house. In the past we would have just hired a consultant, so I believe it saved us money and effort." - Ethanol Facility Facilities, in general, could not point to instances where the use of the Manual had influenced a compliance change. One exception was a facility that had started up after the Manual was published, who said, "Before we started up our plant, we read the Manual and realized that we had missed a permit," which prompted the facility to secure the permit. No facilities could point to any environmental or health outcomes that resulted from the use of the Manual. One facility noted that the main impact on their management processes was that the information in the Manual allowed the facility to do more of their permitting work in-house rather than hiring outside consultants. November 2009 Page 14 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report One EPA interviewee recalled being contacted because of someone reviewing the Manual and having a follow-up question. Survey Findings Overall, survey respondents expressed satisfaction with the information thoroughness of the Manual. Sixteen respondents, or approximately 76 percent, were satisfied or very satisfied with how well the Manual met their needs for understanding compliance information, and five, approximately 24 percent, were neither satisfied nor dissatisfied; none expressed dissatisfaction. One contractor respondent was dissatisfied with the information thoroughness of the Manual, suggesting that the air section should be updated and expanded with more specific details. All other respondents were either satisfied or very satisfied with the information thoroughness (15 respondents, or greater than 70 percent) or were neither satisfied nor dissatisfied (five respondents, or 24 percent). Compliance Data Analysis Table 1. Pre- and Post-Manual Publication Ethanol Violations Informal Violation or Notice Pre-Nov. 2007 48.7% Informal Violation or Notice Post -Nov. 2007 34.6% Formal Violation Pre-Nov. 2007 11.5% Formal Violation Post-Nov. 2007 9.0% N=78 Analysis of EPA's Enforcement and Compliance History Online (ECHO) database provided the above information. ECHO provides an overview of a facility's environmental record under the Clean Air Act (CAA) Stationary Source Program, the Clean Water Act (CWA) National Pollutant Elimination Discharge System (NPDES), and the Resource Conservation and Recovery Act (RCRA), as well as state and federal responses. This information shows that there had been a decrease in informal violations since the Manual was published in November 2007. Prior to November 2007, 48.7 percent of the facilities in Region 7 had received an informal violation, compared to 34.6 percent after publication of the Manual. For formal violations, 11.5 percent of Region 7 facilities received a formal violation before publication of the Manual, compared to 9.0 percent after publication. However, this evaluation cannot conclusively state that environmental compliance increased because of this Manual. The data in ECHO may have been incomplete or lagging and may not have completely shown the enforcement actions taken in all of 2009, and the evaluators did not have access to information on the rate at which inspections were conducted for facilities in the Region on a yearly basis. In addition, this evaluation had incomplete information about whether all of the plants analyzed were in full operation before November 2007 and whether they continue to be in operation at the date of this evaluation. Further, past enforcement actions can influence current compliance with regulations, as can other factorssuch as learning from the mistakes of other facilities, a downturn or upturn in business, or changes in personnel with compliance responsibilities. Finally, this evaluation did not have information about whether all of these facilities received and reviewed the Manual and therefore what, if any, effect it may have had on their compliance behavior. The evaluators also looked more specifically at the facilities that were interviewed for this evaluation and that responded to the survey. Below are the findings of this analysis: One facility had more than 15 informal violations between 2005 and 2007, but only two in 2008 and 2009, all pertaining to air regulations. The facility interviewee did not explicitly credit the Manual with helping improve compliance, but did note that having the information in the Manual would have been November 2009 Page 15 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report helpful when the facility was trying to be in compliance when just beginning operation, because the Manual lays out all the regulations and needs clearly. The interviewee particularly noted that the section on the CAA was helpful, as CAA requirements can be very confusing to facilities. Another facility that responded to the survey had received 15 informal CAA violations between 2007 and 2009. In the survey, the respondent from this facility expressed satisfaction with the CAA information in both the building and operation sections of the Manual. This respondent also suggested that voluntary EPA inspections would be a helpful way to learn about compliance information. This facility indicated that they had received the Manual from a state contact, though did not indicate a timeframe for this interaction. Another facility that responded to the survey had received a formal enforcement action under the CAA earlier in 2009. This respondent expressed dissatisfaction with the information thoroughness of the Manual and requested more specific examples and details. As noted above, this evaluation cannot conclusively state that there is any causal link between these compliance actions and use of the Manual. During discussions with the Region 7 Biofuels Team about the findings of this report, members of the Biofuels Team noted that more consistent tracking of compliance and beyond compliance behavior and targeting assistance based on more thorough compliance information may be a path forward for this sector. Topic Area IV: Improving the Manual How could the Manual be improved to provide effective compliance assistance to ethanol facilities? Overarching Findings Respondents to this evaluation had a number of suggestions for how future versions of Manual could be improved. The suggestions centered around additional content, such as a section on cellulosic ethanol, or around functional content, such as checklists, case studies, call-out boxes, and examples. Interview Findings Most interviewees had some general comments and suggestions for how the Manual could be improved to provide effective compliance assistance, as well as specific suggestions for additions. They offered the following suggested changes and additions to the content: A few contractors requested more examples be provided in the Manual to define terms, such as what constitutes commencing construction and modifying a plant. A contractor suggested including information on how long the permitting process takes, as well as a general discussion of what permits would likely be required, as many operators start the process without any knowledge of the expected permit timeframes. A contractor suggested providing embedded hyperlinks to the Code of Federal Regulations (CFR) Web site so that readers can look up the referring statutes. A contractor suggested moving the Renewable Identification Numbers (RINs) requirements discussion to a later section of the Manual, rather than up-front, since this requirement comes further along in the facility operation process. A contractor noted that some of the permits listed in the Manual did not seem applicable, and three contractors noted other EPA permits and reporting plans applicable to ethanol plants that were not November 2009 Page 16 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report mentioned in the Manual, including: Toxic Substance Control Act (TSCA) Form U Reporting and New sources performance standards (Subpart VVa) (released after the Manual was initially published). A contractor disagreed with the statement found in the Manual that "all ethanol plants will probably generate some quantities of hazardous waste" (page 2-20), and noted that some facilities were focusing on making use of their byproduct waste streams to make value-added products. A facility suggested adding an executive summary to the front of the Manual to make the document more approachable to potential readers. In addition, the evaluators specifically asked interviewees about any suggested changes to the "Who to Contact" section of Appendix A. There was no clear consensus from interviewees on whether this section should provide direct numbers and names to contact people or to general support numbers. Interviewees noted the benefit of having these specific names in knowing whom to contact but noted that staff turnover could cause this section to be out of date sooner than other parts of the Manual. One state interviewee suggested expanding this section to include all applicable ethanol offices in state agencies so that those outside of Region 7 could use the Manual. Interviewees were asked whether there were other areas of information that EPA should consider covering in future versions of the Manual. The following were the suggestions provided for this area: An EPA interviewee noted that issues have arisen in recent years around emissions from fermentors, biochemical oxygen demand loading and storm water ponds, and leak detection. These issues and others that have come to the attention of regulators may be candidates for specific discussions in future versions of the Manual. An EPA interviewee suggested that checklists would be helpful to facilities that have been in operation and were familiar with much of the Manual content. A state interviewee noted that the Manual was geared toward building a new plant, but that any future version could include information about requirements when making substantial modifications to that plant or process (e.g., using a new feed stock, burning fiber). Similarly, two contractors noted that ethanol plants were looking for ways to integrate value-added products to their production stream (e.g., oil extraction), and that including a discussion of requirements that plants should consider would be useful. Related to this discussion, a state compliance staff member suggested a section on byproduct handling. An EPA interviewee suggested adding information about state laws to make the Manual more comprehensive, while noting that it would be more difficult to update that information. Facilities and contractors, similarly, said that ethanol plants were responsible for a number of federal requirements to different agencies, including the U.S. Department of Transportation, the U.S. Department of Homeland Security, the U.S. Army Corps of Engineers, and the Alcohol and Tobacco Tax and Trade Bureau. EPA and state staff as well as contractors noted that future versions could include sections on cellulosic and other emerging types of ethanol. State staff and contractors suggested that a future version of the Manual could include information on what should happen with federal permits when plants halt production for any length of time. In addition, unless there is renewed emphasis on alternative fuels, there may not be as much construction of facilities in the future; any future versions could focus on operation rather than construction of plants. November 2009 Page 17 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report An EPA staff member suggested information on community and emergency response could be valuable for this audience and that EPA should incorporate this information or link to another organization that has resources in this area. Survey Findings The survey asked facilities and contractors what things EPA should consider to improve satisfaction with the Manual as a compliance assistance tool. The following were the suggestions provided in the survey for additional content or functionality: More examples and specific details throughout the text A list of ethanol relevant resources, such as ethanol trade associations A list of contractors or consultants in particular states and the services they provide Checklists and "at a glance" materials and pages In addition, a facility expressed dissatisfaction with the transition between the construction and operation sections of the Manual, but did not elaborate on this response. Another noted that the section on air quality regulations needs updating and improving, but also did not expand on this comment. The survey asked participants how frequently EPA should update the Manual to be helpful to facilities. Ten respondents (48 percent) suggested updating the Manual every three years. The three respondents who chose "Other" wrote in that Region 7 should update the Manual whenever there were regulatory changes (see Figure below). Which update frequency would be the most useful for the Manual? N=21 Other timeframe 14% Every five years 9% I don't intend to refer to the Manual again (0%) November 2009 Page 18 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Topic Area V: Other Compliance Assistance What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? Overarching Findings As previously discussed, evaluation participants thought the Manual was a useful tool for conveying compliance information. However, evaluation participants also considered other venues, such as conferences, EPA workshops, a more interactive Web site, and one-on-one interaction with ethanol facilities to be additional valuable approaches for providing compliance assistance. Interview Findings During the interviews, facilities and contractors were asked about other tools and approaches that EPA could use to present the information that is available in the Manual. Five of the ethanol facility representatives noted that they had attended EPA workshops in the past and found them to be useful sources of information about regulations and implementing them at their facilities. In addition, a few of the contractors and facilities regularly attended conferences; additional or continued EPA presence would be welcome there to share information about new resources or requirements. In addition, interviewees of all categories noted that having an up-to-date Web site that is easy to navigate was a very effective way to convey information. One contractor suggested having an ethanol-specific Web page, either nationally or regionally, to provide links to resources and updates on regulations. Survey Findings The survey asked respondents about what other tools and approaches would be helpful for conveying compliance assistance information. Most noted that compliance assistance had so far been sufficient. Below were the other suggestions from the survey: An email newsletter or listserv with reminders, updates, industry contacts, suggested best practices, and regulatory changes applicable to the industry. Voluntary EPA inspections without fines. Additional workshops and meetings with open discussions between regulators and ethanol plants. November 2009 Page 19 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report IV. Recommendations This evaluation assessed the effectiveness of an ethanol compliance assistance Manual developed by EPA Region 7. The evaluation focused on a set of overarching evaluation questions to inform this assessment. Evaluation participants clearly conveyed that the Manual was useful and valuable to its intended audiences, and that it effectively communicates relevant information for ethanol plants. The following recommendations focus on ways to improve the Manual for any potential future versions and other ways that Region 7 could make this information available to its intended audiences. The evaluators understand that there may be limited resources available for updating the Manual and limited time available for staff to provide other kinds of compliance assistance to ethanol facilities. The evaluators also recognize that the market for ethanol has been shifting dramatically, and that the demand for new facility permits has not returned to the peak it was in when Region 7 first released the Manual. These and other related factors will likely influence demand for the Manual and feasibility of implementing any of the evaluation's recommendations. The below recommendations are organized thematically and not by priority. 1. As a first step, consider whether to tailor the Manual's content and distribution to the national level. The Manual is available to the public on the Region 7 Web site, and as such it is available nationally. However, the contact information in Appendix A is focused on Region 7 only, and EPA's national offices have not posted the Manual on their Web sites. EPA and state compliance staff and contractors noted that the Manual's content was not specific to Region 7, and would be of interest to other Regions with significant ethanol industries. Already, some state agencies such as the Kentucky Department for Energy Development and Independence and the Minnesota Pollution Control Agency have linked to the Manual on their Web sites, noting that it may be of interest to their states' ethanol facilities. The Region 7 Biofuels Team noted that they had written the Manual in such a way that it could be easily adapted to other EPA Regions. The Region 7-specific contact information, for example, was included in an appendix to enable easy substitutions with other Regions' contact information. Sending the Manual to EPA staff in other Regions for posting and distribution would be a simple first step to making this information more widely available, and Regional staff or Headquarters could take the lead on revising the contact Appendix and any other sections that needed another Region's specific information. Staff in the other Regions and at state environmental agencies would likely need to be available to answer questions about the information provided by the Manual, which would require a commitment of staff time in other Regions with significant numbers of ethanol plants. The evaluators recognize that there may not be a clear national EPA office or program to take over responsibility for the current Manual, any updates to the Manual, or any coordination between or among Regions for the Manual. Similarly, other Regions may not have an equivalent biofuels team or group to coordinate the Manual's distribution or incorporate regional contact information. For these reasons, if the Manual were to be used nationally or in other Regions, it may be necessary for the Biofuels Team in Region 7 to identify individual "champions" for the Manual's use and updates at the national and/or regional levels. Some level of ongoing coordination between Regions or between Regions and Headquarters may be needed. November 2009 Page 20 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report 2. Consider additional outreach strategies and targeted audiences. Region 7 undertook a number of activities to advertise the Manual's existence, including a news release, providing postcards at conferences, emailing those who had expressed interest, and highlighting the Manual during media interviews. The Region could continue to advertise the Manual and any future versions with these mechanisms, but could also expand its outreach to consider additional audiences and outreach strategies. The evaluators recognize that Region 7 had limited resources to conduct Manual outreach and advertisement. Allocating at least some resources to outreach would allow the Region to conduct activities such as advertising in trade journals relevant to the ethanol industry. However, Region 7 could also undertake activities that would not require significant resource investment and would be more likely to reach the targeted audiences relatively quickly: Region 7 could develop and maintain an ethanol-industry email listserv that regularly sends out updates and news to subscribers. Region 7 could allow anyone to join or have a moderated listserv of only ethanol plant managers. This approach would be a simple way to keep in contact with interested stakeholders and alert them to any future versions or updated Manual, or to remind them of the Manual's existence, as well as send out other relevant information. When Region 7 staff members visit ethanol plants on site visits, the staff could bring an electronic copy of the Manual or could provide a postcard with the Web link to the Manual. If the Manual is updated with new information, Region 7 could gather a small focus group of three to five ethanol facility contacts by phone or in-person. This group could discuss the best ways to inform ethanol plants about any changes, and whether the industry needs a fully updated Manual with the information, or whether an email or Web site with the changes is sufficient. This approach would allow Region 7 to learn about any venues (e.g., upcoming conferences) that could be key distribution points, and would also allow Region 7 to target its resources strategically. While ethanol plants were the primary targeted audience for this Manual, ethanol-industry contractors could be an effective audience for future outreach. Many of the facilities interviewed said that they use contractors for much of their environmental compliance and permitting work, and rely on these contractors for updates and information on regulation. As there are a smaller number of contractors and consultants working with existing and potential ethanol facilities, making the Manual specifically available to contractors could more directly reach an important audience and perhaps ultimately reach a wider audience. In addition, having contractors review draft Manual updates may be a productive way to ensure that the Manual is helpful to the target audience. 3. Consider making this tool primarily Web-based, with a hard copy Manual as a supplement. Nearly all evaluation participants accessed the Manual on the Internet, and many suggested that the Manual include more Web-based content, such as updates and additional or more current contact information. Transferring the Manual to a current, Web-based version would require some effort, particularly if user interactivity such as topic "drill downs" and links to related topics or contacts, were to be incorporated. Making the content Web-based, however, would make the Manual more accessible and useful to many users and likely easier to maintain and update by EPA. In addition, Internet search engines could find specific sections more easily, and Region 7 could implement Web feeds that would allow users to subscribe and be notified of any changes.12 A version formatted for printing could still be 12 "Web feeds" are a format for providing users with frequently updated content. Content distributors (in this case, Region 7) develop a "feed" and allow users to subscribe to it. Users are alerted through a "feed reader" or through November 2009 Page 21 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report available on the Web for readers who wanted to download it for their office reference library or for EPA and state compliance staff or others who would want to take hard copies to facilities they visit. The below are some ways that Region 7 could implement this recommendation: Region 7 could simply upload the content onto a series of HTML Web pages. This approach would allow EPA staff to make changes quickly to the content of the Manual, without republishing the document entirely. Examples of this approach at EPA include the EPA Lean in Government Starter Guide13 and the EPA Guide for Industrial Waste Management.14 Region 7 could upload the Manual to the Web in a wiki-format,15 which would allow EPA staff to edit the content even more easily than HTML pages, and could allow other non-EPA users to edit the content. This ease of updating can help ensure that content is up-to-date and relevant to its intended audiences. Region 7 could upload the content to an EPA-hosted or third-party hosted wiki and then designate through assigned log-ins which groups (e.g., EPA staff, state compliance staff, facility contacts) are allowed to contribute edits to the document and create discussions about specific pages or sections. This approach could allow a greater cross-section of stakeholders to contribute to improving the Manual, and could result in a more valuable resource for ethanol plants looking for compliance assistance information. If Region 7 decides to go this route to make the Manual Web- based, they can look to the lessons learned of similar efforts at EPA and other federal agencies. For example, EPA's Watershed Central Wiki is a Web site based on the MediaWiki platform that is open to EPA employees and identified partners to provide information on watershed resource management best practices, organizations, and plans.16 Region 7 could develop a Web site that displays the content of the Manual on a series of HTML or Wiki Web pages (discussed above), and also links to more dynamic content and resources internal and external to EPA, such as upcoming meetings or relevant conferences. For example, the EPA Office of Wastewater Management has developed a Web site on NPDES (National Pollutant Discharge Elimination System) permits.17 It links not only to compliance information, but also to relevant new publications and training materials, discussions of current issues, and links to upcoming meetings and conferences. While incorporating and updating this material would require more staff time and resources on a continuous basis, it could become a frequently used and valuable resource for ethanol plants and their environmental managers. 4. Update with current information on regulations. If a future version of the Manual is published, it could incorporate the latest information on applicable federal environmental regulations, permits, plans, and approvals for ethanol plants. Alternatively, a stand-alone supplement to the Manual that focuses solely on recent changes and updates could be created. If the Manual was Web-based (see Recommendation 3), regular maintenance would ensure that all of these components are kept current. If the Manual continued to be hard-copy based, then having their email when new content is available. An example of this is EPA's Office of Public Affairs news release pages, which alerts users whenever a new press release is issued; see http://www.epa.gov/newsroom/rssfeeds.htm. 13See http://www.epa.gov/lean/starterkit/index.htm 14See http://www.epa.gov/waste/nonhaz/industrial/guide/index.htm 15 A wiki is a Web site that allows easy creation and editing of content for users. Wikis are often used for collaborative editing of a document that has content that changes frequently and/or could benefit from multiple perspectives. Wikipedia (http://www.wikipedia.org) is one of the best known wikis. 16 See https://wiki.epa.gov/watershed/index.php/Main_Page; Access is limited to EPA employees and identified partners. 17 See http://cfpub.epa.gov/npdes/index.cfm November 2009 Page 22 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report links within the Manual to an EPA Web page that provides real-time updates to the Manual content would keep the information current for ethanol facilities and contractors looking for updates. 5. Create an executive summary to encapsulate the main points of the Manual. A few participants in this evaluation noted that the length of the Manual could detract potential readers from reviewing the Manual; however, no participants noted opportunities for shortening the Manual.18 In addition, several participants thought that a short executive summary or equivalent synopsis would give readers an overview of the content and any potential sections of interest without taking away any of the content within the Manual. There is currently a front section ("What is the purpose of this manual?") that provides two paragraphs on the purpose of the Manual and some disclaimers on what the Manual does not cover; however, this section is not an executive summary in the way that readers expect. Similarly, the Introduction covers three topics (requirements for renewable fuels, the renewable fuel standard program, and a brief overview of the ethanol production process), but does not include a list of the laws that apply when constructing, modifying, or operating ethanol plants or any related summary information. Creating a new executive summary or overview section to summarize the main topics of each of the chapters could be one way to meet the stated need of respondents. 6. Incorporate more "at a glance" information. Expanded "at a glance" information, such as lists of resources, call-out boxes, checklists (e.g., on the steps to take), and calendars can be integrated throughout the Manual to highlight particularly important information. Respondents to this evaluation indicated that they would like more of this kind of content so that they can quickly refer to information of interest. The evaluators recognized that the Biofuels Team intentionally avoided creating new guidance with the Manual, and as such that there may be reason not to create action or requirements lists that could constitute guidance. If this has not been done already, it may be worth exploring the feasibility of drawing from existing guidances to develop "how to" checklists or other short reference lists either at the regulation, media, and/or cross-media levels. Similarly, a generic schedule that a plant could modify and download could be very useful. Such a schedule could, for example, indicate a starting date and include the deadlines (e.g., date + 90 days) for the related required procedures, submissions, and other information. It may be that plants or their contractors already have these tools in paper or electronic format and may be willing to share them for use by EPA for this purpose. As mentioned earlier in this report, one facility interviewee developed a calendar for this purpose after reading the Manual. Several participants raised the idea of listing resources beyond the contacts already included in Appendix A. Some said that a list of contractors would be of interest, though they also said that caution may need to be exercised in doing so to avoid perceived or actual conflicts of interest. Some thought that there were precedents for listing contractors and that there would be no problems in doing so as long as there were multiple contractors listed. Others thought that directing parties to external Web sites that already list contractors would be a relatively easy way to achieve this change, though it is not clear that there is such a resource. Other resources that Region 7 could consider could be lists of facilities that would be willing to discuss their experience with others and contacts from other regulatory agencies, such as the Occupational Safety and Health Administration (OSHA). 18 One way to reduce the manual's size would be to reformat it with fewer pictures (though the pictures make it a much more visually pleasing document), reduce the large margins, and modify the spacing. Converting the Manual to a Web-based format may or may not assist with length depending on how the Web site is designed. This would not serve as a substitute for including an executive summary but would likely help with the initial impression voiced by several participants that the Manual seems very long. November 2009 Page 23 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report In developing the Manual, the Biofuels Team has considered about whether a "flip book" of key issues and important points would be feasible. The evaluators think that anything along these lines, regardless of the format (e.g., flip book, standard paper format, electronic spreadsheet) would be useful for readers. 7. Provide more specific examples and case studies. Participants commented that the information in the Manual was sometimes vague or too general and that specific examples would be helpful for understanding. For example, providing examples of what constitutes a modification to a building would be helpful to a reader wanting to understand what is applicable to their building. More narrative discussion or case studies, either real or hypothetical, can also help a reader relate their facility to the regulations discussed within the Manual. 8. Expand content to include regulatory information about topics of interest to the target audience, such as cellulosic ethanol production and byproduct diversion into other products. There were many suggestions for specific content to add to the Manual for future versions (see the findings on Topic Area IV: Improving the Manual). Cellulosic ethanol production and byproduct diversion into other products were mentioned most often by the state compliance staff, facilities, and contractors who participated in the evaluation. Cellulosic ethanol production is expected to increase as producers look to different feedstocks to take advantage of new incentives, research findings and changing market conditions, and facilities are interested in the different federal environmental regulations that may be applicable. In addition, during this evaluation, ethanol facilities indicated that they were looking for ways to take the byproducts of ethanol production, make them into value-added products, increase profit margins, and decrease waste disposal. Facilities may find value in a discussion in the Manual of regulations and requirements to consider when changing operations in these ways. 9. Expand information on beyond compliance efforts and best practices. Although the Manual is focused on current federal environmental regulations and rules, a section on how a facility could go beyond compliance and implement best practices may be of interest to some facilities. In particular, newer or smaller facilities may not know about best practices that could provide environmental, safety and/or economic benefits; having this information in one place could be useful. Currently, the Manual has a section in the construction chapter about the Pollution Prevention Action and refers to the EPA Pollution Prevention Web site.19 Region 7 could evaluate whether this is sufficient information or whether expanding this section with additional best practices could be a way to incorporate more beyond compliance information for the audience. The evaluators recognize, however, that providing beyond compliance information is a secondary priority following the Manual's focus on compliance assistance. 10. Apply the lessons from this evaluation to the Region's other compliance assistance and beyond compliance efforts. The evaluators understand that Region 7's compliance assistance activities are largely integrated into its broader compliance-enforcement program. The Region has a compliance assistance coordinator who serves the regional media program staff as a policy disseminator and a data quality control check for measuring activities. The technical and enforcement staff in the other regional programs incorporate the actual compliance assistance activities into their other routine actions. More broadly, the Region focuses on conducting inspections, following up on potential violations, and, in general, working within the traditional compliance-enforcement framework that has been the hallmark of EPA's compliance approach for over 30 years. Region 7, like other Regions, also oversees the compliance enforcement work of the 1Q See http://www.epa.gov/p2/ November 2009 Page 24 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report Region 7 states and reports back to EPA Headquarters Office of Enforcement and Compliance Assurance (OECA) and on its activities and progress through routine reporting channels. Compliance assistance is, in other words, primarily an "add on" to more "core" programmatic activities and responsibilities. Within this broader construct, the Region conducts both ad-hoc and routine periodic compliance assistance work. The Manual is an ad-hoc example that the Region took on after a dramatic shift in the number of new permit requests for ethanol facilities as explained in this report. In another instance, Region 7 is focusing on improving compliance with a new lead (Pb) rule, which was promulgated in April 2008. This rule requires workers to follow lead-safe work practice standards to reduce potential exposure to dangerous levels of lead during renovation and repair activities.20 The Region has been conducting outreach with thousands of contractors and citizens who renovate, repair, or repaint older buildings that may have lead-based paint to support compliance with the new rule. The Region also discusses compliance assistance activities each quarter with the Region 7 states. At these meetings, the state and regional staff take turns sharing their compliance assistance activities among discussions of other activities for collaboration. Within this context, the evaluators, in consultation with Region 7 staff, have identified some options for applying the lessons learned from the evaluation of the ethanol compliance assistance manual to the Region's broader compliance assistance activities. These options, which can serve as the basis for further discussion within the Region, are as follows: i. Determine whether the Region would like to initiate more compliance assistance activities like the ethanol compliance assistance manual that are proactive in nature and intended to help raise awareness amongst a targeted regulated community that has been identified as needed additional support. Doing so would require additional planning and coordination and therefore the "blessing" of senior managers who would deploy staff resources for this purpose. In addition, there may need to be further exploration of the relationship between the compliance assistance work and the more customary interactions that compliance inspectors have with the regulated community to clarify respective roles and responsibilities. ii. Consider options for cross-media compliance assistance integration like the Biofuels' Team effort on the ethanol manual. Evaluation participants expressed their appreciation for having all major regulations described in one place instead of media-by-media and program-by-program. iii. Consider additional opportunities for coordinating with, as well as learning from, the Region 7 state compliance programs. Although Region 7 already works hand-in-hand with the state compliance enforcement programs, there are likely opportunities to be more proactive in terms of identifying potential compliance gaps or assistance opportunities, learning about planned or ongoing compliance activities that the states are conducting and coordinating on compliance assistance implementation. There are likely efficiencies and economies of scale for doing so; for instance, in providing regional compliance assistance materials (such as the Manual) to state compliance staff and vice versa, or in having strategic planning meetings intended to design and coordinate Region-wide compliance activities for the upcoming years (this would be materially different from the current update-oriented quarterly meetings). iv. Take advantage of Internet-based tools and options. Tools such as Webinars, interactive guides, Wikis, and topical drill-down Web sites (such as the one discussed in the context of the ethanol 20 See: http://www.epa.gov/region7/citizens/lead.htm November 2009 Page 25 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Final Report compliance assistance manual) provide opportunities for getting compliance information out to a wide range of people quickly. In general, these ideas and options will depend on whether Region 7 wishes to invest in more targeted compliance assistance on top of the current compliance-enforcement activities. There are likely to be needs related to existing or new rules where additional compliance assistance focus would help to raise awareness and support compliance before any inspectors contact regulated entities. Perhaps Region 7 is already exploring such options in relation to other sector outreach, or perhaps this is an area that is worth exploring. As a first step, Region 7 may wish to identify a point person or small group of individuals to further explore the options for either following up on recommendations related to the ethanol compliance assistance manual itself, or on taking a next step on additional targeted compliance assistance that can benefit from the lessons gained from the Manual evaluation. It is likely that a targeted scoping exercise could result in a short set of viable options and recommendations for future "value added" compliance assistance that Region 7, its states, and the Regional regulated community could all utilize to support improved understanding of and compliance with the law. November 2009 Page 26 ------- ?/ERA United States Environmental Protection Agency Office of Policy, Economics and Innovation (1807T) November 2009 EPA-100-R-10-001 Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Table of Contents Appendix A. Customer Satisfaction Survey 1 Appendix B. Customer Satisfaction Survey Results 10 Appendix C. Interview Guide for Ethanol Facilities 19 Appendix D. Interview Guide for Ethanol Contractors 22 Appendix E. Interview Guide for State Compliance Staff 24 Appendix F. Interview Guide for EPA Staff 26 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices EPAICRNo. 1711.05 OMB Control No. 2090-0019 EXPIRATION DATE: 8/31/2009 Appendix A. Customer Satisfaction Survey COMPLIANCE ASSISTANCE MANUAL FOR ETHANOL FACILITIES Survey of Customer Satisfaction Thank you for participating in this survey so that EPA can serve you more effectively. Assuming you have already read the compliance assistance manual, Environmental Laws Applicable to Construction and Operation of Ethanol Plants, we anticipate that the main part of this survey will take approximately 10-20 minutes and the optional questions at the end will take approximately an additional 10 minutes. Your participation will help EPA improve ethanol facilities' satisfaction with this manual. You may also take this survey on-line at the following website: http://www.survevmonkey.eom/s.aspx?sm=2uWpdtMi8JSWiH 2bkMbQ35g 3d 3d. Submitting your responses online will take the same amount of time and will help us to analyze the results and provide feedback to EPA more quickly. If you choose to fill out the paper survey instead, please return your responses by July 10, 2009 in the enclosed self-addressed stamped envelope. Note about respondent confidentiality: The contractors evaluating the ethanol facilities' and contractors' satisfaction with the compliance assistance manual, Environmental Laws Applicable to Construction and Operation of Ethanol Plants, are gathering basic information to better understand how useful different types of customers have found the Manual to be. Informational responses will be generalized and made anonymous for the final results and no facility or contractor names or other identifiers will be provided to EPA as part of the results, unless a facility or contractor explicitly indicates at the end of this survey that they would like their responses to be identified. '-.' I ;:M|, M | M ',( LAWS Ethanol Plants T» m-a-CKH.'! jV .\ ±^!L .'->,% Industrial Economics, Incorporated (http://vwvw.indecon.com/) and Ross & Associates Environmental Consulting, Ltd. (http://www.ross-assoc.com) are the contractor team. For more information, please contact Anna Williams (anna.williams@.ross-assoc.com) or Amy Wheeless (amv.wheeless@.ross-assoc.com) by email or at (206) 447- 1805. November 2009 Page 1 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices I. PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER 1. Prior to receiving this questionnaire, were you aware that EPA developed the Compliance Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available? o Yes O No 2. Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities, published by EPA in November 2007? If No, please skip to Question 17. O Yes O No 3. Please check the method below that indicates how you learned about the Manual O Postcard distributed from EPA 0 Meetings / conferences (e.g., O EPA news release 0 Other correspondence O EPA radio or TV interview 0 Other method (please specify) O O EPA Web site State contact 0 Don't know/don't remember FEW workshop, from EPA CenSARA's National Environmental Biofuels Conference) If other: 4. Why were you interested in receiving the Manual? (check all that apply) 00 00 Wanted to learn more Wanted to learn about General reference Other about applicable Federal applicable Federal laws for laws for construction of operation of facilities facilities If other: November 2009 Page 2 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Optional response: O I received the Manual without asking for it (check here if applicable) PART 2: ACCESSING THE MANUAL 5. Please check the method below that indicates how you accessed the Manual. O O O O O Downloaded the Received a Received a CD Manual from the Compact Disc (CD) from a state EPA Web site from EPA contact Received a CD Received a hard from another copy from EPA contact outside of my facility O O O Received a hard Received a hard Other method copy from a state copy from a (please specify) contact different (not state/EPA) contact outside of my facility If other: 6. Please rate how satisfied you were with the ease of accessing the Manual. 000000 Very Dissatisfied Neither Satisfied Very Satisfied No Opinion Dissatisfied Dissatisfied nor Satisfied Please describe any ways you think EPA could improve access to the Manual (optional): 7. Have you read any part of the Manual? If No, please skip to Question 17. If Yes, please continue to Part 3 of the survey. O O Yes No November 2009 Page 3 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices PART 3: STRUCTURE AND READABILITY OF THE MANUAL 8. Please rate your satisfaction with the Manual's readability. 0000 Very difficult to Difficult to read Neither easy read nor difficult to read o Somewhat easy Very easy to to read read o No opinion 9. Please rate the ease of navigating through the Manual to get to areas of interest. 0 Very difficult to navigate to areas of interest 0 Difficult to navigate to areas of interest 10. How would characterize the O Significantly too little O Slightly too little 0 Not easy, but not difficult to navigate to areas of interest 0 Easy to navigate to areas of interest amount of information presented O Right Amount O Slightly too much 0 Very easy to navigate to areas of interest in the Manual? O Significantly too much 0 No opinion O No opinion We welcome any comments on your response: IV. PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL 11. a. Please rate your satisfaction with how well the Manual met your needs for understanding compliance information. Very Dissatisfied Dissatisfied O O Neither Dissatisfied nor Satisfied O Satisfied O Very Satisfied O No Opinion O November 2009 Page 4 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices b. Please rate your satisfaction with how clear the information provided in the Manual was on where to go for more information. Neither Dissatisfied nor Very Very Dissatisfied Dissatisfied Satisfied Satisfied Satisfied No Opinion 000 000 c. Please rate your satisfaction with the information thoroughness of the Manual. Neither Dissatisfied nor Very Very Dissatisfied Dissatisfied Satisfied Satisfied Satisfied No Opinion o o o o o o We welcome any comments on your response: Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If so, we welcome any explanation and suggestions for improving your satisfaction with that section or chapter below. OYes ONO 12. What else should EPA consider to improve your satisfaction with the Manual as a compliance assistance tool? November 2009 Page 5 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 13. Please rate your overall satisfaction with the Manual as a compliance assistance tool. Very Dissatisfied Dissatisfied o o Neither Dissatisfied nor Satisfied O Very Satisfied Satisfied No Opinion O O O V. PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE 14. If EPA were to update the Manual, which update frequency would be the most useful? O Annually If other: O Every 3 years O Every five years O Other timeframe (please specify) O I don't intend to refer to the Manual again 15. What other tools and approaches would you prefer for sharing compliance assistance information concerning ethanol facilities? VI. FINAL QUESTIONS! 16. What category best describes your position? O O Facility manager Facility compliance officer O o Other facility role Contractor (please specify) If other: 17. Can we identify your facility or company (by name) as having participated in this survey? OYes ONO November 2009 Page 6 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Facility or Company Name (optional) PRIMARY POINT PERSON (OPTIONAL) If you are willing to provide the name of a primary point person who could be contacted by the contractors with any follow up questions or correspondence, please list that person here. Name: Phone Number: E-mail Address: Thank you for participating in this survey! Please return your responses using the enclosed self-addressed stamped envelope by July 10, 2009. November 2009 Page 7 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Optional Section: if you would like to provide more specific feedback on each Section of the Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer the following questions. 1. Please rank your overall satisfaction with the ease of understanding, information thoroughness, and overall usability of each of the Manual's sections: Neither Dissatisfied Very nor Very No Dissatisfied Dissatisfied Satisfied Satisfied Satisfied Opinion Introduction Section 1.1: Requirement Q O O O O O for Renewable Fuels Section 1.2: Renewable Q O O O O O Fuel Standard Program Section 1.3: Brief Overview of Ethanol O O O O O O Production Process Chapter 1: What Laws Apply When I'm Constructing or Modifying a Plant Section 1.1: National Environmental Policy Act Section 1.2: Clean Air Act Section 1.3: Clean Water Act Section 1.4: Safe Drinking Water Act Section 1.5: Pollution Prevention Act Chapter 2: What Laws Apply Section 2.1: Clean Air Act Section 2.2: Emergency Planning and Community Right to Know Act Section 2.3: Clean Water Act Section 2.4: Safe Drinking Water Act O O O O O to Operating O O O O O O O O O an Ethanol O O O O O O O O O Plant? O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O November 2009 Page 8 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Neither Dissatisfied Very nor Very No Dissatisfied Dissatisfied Satisfied Satisfied Satisfied Opinion Section 2.5: Resource Conservation and Recovery Act - Hazardous Waste Appendices Appendix A (First Half): Summary of Laws Pertaining to Ethanol Production and Contacts Appendix A (Second Half): Contact Information Appendix B: Emergency Planning Appendix C: Clean Air Act Section 112-R Requirements Appendix D: Definitions Of Acronyms Appendix E: The NEPA Process o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o Do you have any specific suggestions for improving the usefulness of any of the Manual sections? (optional) If you have not done so already, please remember to fill out the final section of the main part of the survey starting with "Part 4: Your Overall Satisfaction with the Manual" on page 3. Thank you! Burden Statement: Public reporting burden for this collection of information is estimated to average fifteen (15) minutes per response, including the time for reviewing instructions, gathering information, and completing and reviewing the collection of information, with a possible ten (10) additional minutes for the optional section at the end. Send comments on the Agency's need for this information, the accuracy of the provided burden estimates, and any suggestions for reducing the burden, including the use of automated collection techniques to the Director, OEI Collection Strategies Division, United States Environmental Protection Agency (Mail Code 2822T), 1200 Pennsylvania Avenue, Washington, DC 20460; and to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17th Street NW, Washington, DC 20503, Attention: Desk Officer for EPA. Include the EPA ICR number (1711.04) and the OMB control number (2090-0019) in any correspondence. November 2009 Page 9 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Appendix B. Customer Satisfaction Survey Results Note: Questions that asked for optional contact information are not listed here, nor are any participant comments that indicated which surveys responded to the survey. PART 1: INFORMATION ABOUT YOU AS OUR CUSTOMER 1. Prior to receiving this questionnaire, were you aware that EPA developed the Compliance Assistance Manual for Ethanol Facilities (Manual) and that the Manual was publically available? Response Response Percent Count Yes 72.7% 16 No 27.3% 6 answered question 22 skipped question 2 2. Have you received (or downloaded) the Compliance Assistance Manual for Ethanol Facilities, published by EPA in November 2007? If No, please skip to Question 17. Response Response Percent Count Yes 100.0% 22 No 0.0% 0 answered guest/on 22 skipped question 2 3. Please check the method below that indicates how you learned about the Manual Response Response Percent Count Postcard distributed from EPA 13.0% 3 EPA news release 4.3% 1 EPA radio or TV interview 0.0% 0 EPA Web site 21.7% 5 State contact 17.4% 4 Meetings / conferences (e.g., FEW workshop, CenSARA's National Environmental Biofuels 13.0% 3 Conference) Other correspondence from EPA 8.7% 2 Other method (please specify) 17.4% 4 Don't know/don't remember 4.3% 1 Other (please specify) From corporate office From this survey and this consulting firm 4 This questionnaire From Ross & Associates November 2009 Page 10 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices answered question skipped question 23 1 4. Why were you interested in receiving the Manual? (check all that apply) Response Percent Wanted to learn more about applicable Federal laws for construction of facilities Wanted to learn about applicable Federal laws for operation of facilities General reference Other (please specify) I received the Manual without asking for it answered question skipped question 22.7% 63.6% 54.5% 4.5% Response Count 14 12 1 3 22 2 PART 2: ACCESSING THE MANUAL 5. Please check the method below that indicates how you accessed the Manual. Downloaded the Manual from the EPA Web site Received a Compact Disc (CD) from EPA Received a CD from a state contact Received a CD from another contact outside of my facility Received a hard copy from EPA Received a hard copy from a state contact Received a hard copy from a different (not state/EPA) contact outside of my facility Other method (please specify:) Received electronic copy from corporate office Electronic copy from state contact Received link from Ross & Associates Emailed to me by EPA contact answered question skipped question Response Percent 73.7% 0.0% 0.0% Response Count 14 0 0 0.0% 15.8% 5.3% 5.3% 3 1 19 5 6. Please rate how satisfied you were with the ease of accessing the Manual. Very Dissatisfied Dissatisfied Neither Dissatisfied nor Satisfied Satisfied Very Satisfied Response Percent 0.0% 0.0% 4.3% 43.5% 39.1% Response Count 0 0 1 10 9 November 2009 Page 11 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices No Opinion ^^ 13.0% answered question skipped question 23 1 Please describe any ways you think EPA could improve access to the Manual (optional): No Answers 1. Have you read any part of the Manual? If No, please skip to Question 17. If Yes, please continue to Part 3 of the survey. Response Response Percent Count 91.3% 21 Yes No ^^ 8.7% answered question skipped question 23 1 PART 3: STRUCTURE AND READABILITY OF THE MANUAL 8. Please rate your satisfaction with the Manual's readability. Very difficult to read Difficult to read Neither easy nor difficult to read Somewhat easy to read Very easy to read No opinion 9. Please rate the ease of navigating through the Manual to get to areas of interest. Very difficult to navigate to areas of interest Difficult to navigate to areas of interest Not easy, but not difficult to navigate to areas of interest Easy to navigate to areas of interest Very easy to navigate to areas of interest No opinion Response Percent 0.0% 0.0% 4.8% 47.6% 47.6% 0.0% answered question skipped question Response Count 0 0 1 10 10 0 21 3 Response Percent 0.0% 0.0% Response Count 0 0 23.8% 57.1% 19.0% 0.0% answered question skipped question 12 4 0 21 3 November 2009 Page 12 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 10. How would characterize the amount of information presented in the Manual? Significantly too little Slightly too little Right Amount Slightly too much Significantly too much No opinion Response Percent 0.0% 38.1% 57.1% 4.8% 0.0% 0.0% answered question skipped question Response Count 0 8 12 1 0 0 21 3 We welcome any comments on your response: No Answers I IV. PART 4: YOUR OVERALL SATISFACTION WITH THE MANUAL 1 11. a. Please rate your satisfaction How well the Manual met your needs for understanding compliance information How clear the information provided in the Manual was on where to go for more information The information thoroughness of the Manual Very Dissatisfied 0 (0%) 0 (0%) 0 (0%) with the following: Neither Dissatisfied Di^atisfied Satisfied 0 (0%) 5 (23.8%) 0 (0%) 6 (28.6%) 1 (4.8%) 5 (23.8%) Satisfied 13 (61.9%) 1 1 j. j. (52.4%) 13 (61.9%) Very Satisfied 3 (14.3%) 4 I (19.0%) 2 (9.5%) No Opinion 0 (0%) 0 (0%) 0 (0%) answered quest/on 21 skipped question 3 We welcome any comments on your response Should have provided some more details. November 2009 Page 13 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Were there any sections of or chapters of the Manual you were particularly dissatisfied with? If so, we welcome any explanation and suggestions for improving your satisfaction with that section or chapter below. Response Percent Yes 9.5% No 90.5% answered question skipped question Response Count 2 19 21 3 We welcome any comments on your response: The section on air quality regulations needs improvement and updating. Construction to operations transition What else should EPA consider to improve your satisfaction with the Manual as a compliance assistance tool? A section targeted to the regulators (EPA/states) where the ethanol plants can provide input on details of plant operations and the importance of operational flexibility. Good overviews. Wish it was more thorough or provided examples. Is there a list of resources or contractors/consultants that could be provided for each state and what services they provide? I really can't think of anything, this was a very comprehensive manual. Include checklists, one-page "at-a-glance" tables, etc. The more specific and less vague it is, the more valuable a resource it will be. No suggestions at this time. Nothing 12. Please rate your overall satisfaction with the Manual as a compliance assistance tool. Answer Options Very Dissatisfied Dissatisfied Neither Dissatisfied nor Satisfied Satisfied Very Satisfied No Opinion Response Percent 0.0% 0.0% 14.3% 76.2% 9.5% 0.0% answered question skipped question Response Count 0 0 3 16 2 0 21 3 November 2009 Page 14 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices PART 5: OTHER SUGGESTIONS FOR FUTURE COMPLIANCE ASSISTANCE 13. If EPA were to update the Manual, which update frequency would be the most useful? Answer Options Annually Every 3 years Every five years Other timeframe (please specify) I don't intend to refer to the Manual again Other (please specify) When regulations change Any time there has been a major regulatory change As often as changes are made answered question skipped question Response Percent 28.6% 47.6% 9.5% 14.3% 0.0% Response Count 6 10 2 3 0 21 3 14. What other tools and approaches would you prefer for sharing compliance assistance information concerning ethanol facilities? Yearly EPA meetings with industry - compliance/update seminars Think it has been good so far. Workshops/meetings with open, unbiased, and frank discussions between regulators and ethanol plants. Email listserv for ethanol industry contacts that sends regulatory/policy changes applicable to the industry or NAICS code. Send industry compliance advisories that give helpful info on what are some compliance concerns along with any compliance assistance info or best practices. An associated question and answer document to is mainted so you could access it to see if there are any changes to the manual. EPA inspection without fines (voluntary) Email updates/newsletters An email newsletter with reminders, updates and new information would be helpful. new regulations and laws that come up every year No suggestions. Website and manual are both sufficient in my opinion. VI. FINAL QUESTIONS! 15. What category best describes your position? Answer Options Facility manager Facility compliance officer Other facility role (please specify) Contractor Response Percent 19.0% 61.9% 14.3% 4.8% Response Count 4 13 3 1 November 2009 Page 15 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices If other: EH&S Manager (2) Consultant Environmental engineer/EHS manager Environmental, Health and Safety Coordinator answered question skipped question 21 3 16. Can we identify your facility or company (by name) as having participated in this survey? Answer Options Yes No Response Percent 52.2% ^^ 47.8% answered question skipped question Response Count 12 11 23 1 Optional Section: if you would like to provide more specific feedback on each Section of the Introduction, Chapter 1, Chapter 2, and the Appendices that you have reviewed, please answer the following questions. 1. Please rank your overall satisfaction with the ease of understanding, information thoroughness, and overall usability of each of the Manual's sections: Answer Options Section I.I: Requirement for Renewable Fuels Section 1.2: Renewable Fuel Standard Program Section 1.3: Brief Overview of Ethanol Production Process Answer Options Section 1.1: National Environmental Policy Act Section 1.2: Clean Dissatisfied 0(0%) 0 (0%) Neither Dissatisfied «-£"- Satisned ^sfTed Option (28.6%) 1 (14-3%) (42.9%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (28.6%) Satisfied 1 (14.3%) 1 (14.3%) 2 (28.6%) °(0%) (28.6%) Neither Dissatisfied «-??« Satisfied JSL Op^on 0 (0%) questic skipped question Very (28.6%) answered question 7 17 0(0%) 0 (0%) Satisfied 0(0%) 1 (14.3%) (71.4%) °(0%) (28.6%) November 2009 Page 16 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Air Act Section 1.3: Clean Water Act Section 1.4: Safe Drinking Water Act Section 1.5: Pollution Prevention Act Answer Options Section 2.1: Clean Air Act Section 2.2: Emergency Planning and Community Right to Know Act Section 2.3: Clean Water Act Section 2.4: Safe Drinking Water Act Section 2.5: Resource Conservation and Recovery Act - Hazardous Waste Answer Options Appendix A (First half): Summary of Laws Pertaining to Ethanol Production and Contacts Appendix A (Second half): Contact Information Appendix B: Emergency Planning Appendix C: Clean Air Act Section 112-r requirements Appendix D: Definitions of 0 (0%) 0 (0%) 0 (0%) Very Dissatisfied 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Very Dissatisfied 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Dissatisfied 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Dissatisfied 1 (14.3%) 0 (0%) 1 (14.3%) 0 (0%) 0 (0%) 1 (14.3%) 3 (42.9%) 1 (14.3%) Neither Dissatisfied nor Satisfied 1 (14.3%) 1 (14.3%) 1 (14.3%) 1 (14.3%) 1 (14.3%) Neither Dissatisfied nor Satisfied 1 (14.3%) 2 (25.0%) 2 (28.6%) 3 (42.9%) 1 (14.3%) (42.9%) (14.3%) (28.6%) 222 (28.6%) (28.6%) (28.6%) (28.6%) ° (0%) (28.6%) 4 2 (57.1%) °(0/o) (28.6%) answered question 7 skipped question 17 Satisfied Satisfied Opinion 6 fQC -JO/ \ 0(0 /o) 0(0/0) (O3./ /O) 5 (85.7%) (71.4%) (14.3%) 0(°°/o) (71.4%) ° (0%) (14.3%) (71.4%) (14.3%) °(°0/0) answered question 7 skipped question 17 -_,.. f. . Very No satisfied Satisfied opinion 2(28.6%) (4230/o) 0(0%) 2(25.0%) (5040%) 0(0%) 4 (57.1%) 0 (0%) 0 (0%) 4 (57.1%) 0 (0%) 0 (0%) 5(71.4%) M * 0(0%) November 2009 Page 17 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Acronyms Appendix E: The NEPA Process 0(0%) 0(0%) 4(57.1%) 3(42.9%) 0(0%) 0(0%) answered question 8 skipped question 16 Do you have any specific suggestions for improving the usefulness of any of the Manual sections? (optional) Answer Options Yes No If yes, please provide: Response Percent 0.0% 100.0% answered question skipped question Response Count 0 8 0 8 16 November 2009 Page 18 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Appendix C. Interview Guide for Ethanol Facilities A NOTE TO INTERVIEWEES A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf If you have an questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206-447-1805 or by email at anna.williams@ross-assoc.com or amy.wheeless@ross-assoc.com. TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? 1. Where did you obtain the Manual? 2. How could distribution of or access to the Manual be improved? TOPIC AREA II: EFFECTIVENESS OF THE MANUAL Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? 1. How readable and comprehendible is the Manual? 2. How comprehensive is the Manual in its scope? 3. What was most/least useful or informative to you in the Manual? 4. Is the contact information provided in the Appendices helpful? How could it be improved? 5. How has your facility used the Manual? If your facility has not used the Manual, why not? 6. What regulatory actions did your facility take, or intend to take, as a result of reviewing the Manual? November 2009 Page 19 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 7. What process or management changes did your facility make or intend to make as a result of reviewing the Manual? 8. Did your facility make any changes in constructing or establishing the plant as a result of reviewing the Manual? TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? 1. After reviewing the Manual, do you feel more aware of the applicable Federal regulations? 2. Are there still federal laws or regulations that you do not understand after reading the Manual? 3. Have there been any changes in your compliance results since reviewing the Manual? 4. What "beyond compliance" measures has your facility made as a result of using the manual? 5. Are there specific environmental health outcomes (e.g. amount of waste generated) that have resulted from use of the Manual? 6. Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that have resulted from this Manual, or are you aware of any potential effects on human health? TOPIC AREA IV: IMPROVING THE MANUAL Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol facilities? 1. What changes would you recommend to the Manual to improve its readability and navigation? 2. What could be changed about the Manual to make it more thorough and comprehensive and to make it more effective as a compliance tool? 3. Are there topics you would like more or new information about in a future Manual? 4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences, or make facilities and other stakeholders aware that an updated Manual is available? 5. Do you have any other suggestions for improving the Manual? TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE Overarching Question: What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? November 2009 Page 20 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 1. What are other ways you find information about regulations and applicable laws? 2. What are other ways EPA could present the information in the Manual to its targeted audiences? (e.g. workshops, interactive Web site) FINAL QUESTION 1. Do you have any final "closing thoughts" or recommendations regarding the overarching questions, sub questions or the compliance assistance Manual in general? Thank you for participating! November 2009 Page 21 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Appendix D. Interview Guide for Ethanol Contractors A NOTE TO INTERVIEWEES A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf If you have an questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206- 447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com. TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? 1. Where did you obtain the Manual? 2. How could distribution of or access to the Manual be improved? TOPIC AREA II: EFFECTIVENESS OF THE MANUAL Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? 1. How readable and comprehendible is the Manual? 2. How comprehensive is the Manual in its scope? 3. What was most/least useful or informative to you in the Manual? 4. Is the contact information provided in the Appendices helpful? How could it be improved? TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? 1. After reviewing the Manual, do you feel more aware of the applicable Federal regulations? November 2009 Page 22 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 2. Are there still federal laws or regulations that you do not understand after reading the Manual? 3. Are there specific environmental health outcomes (e.g. amount of waste generated) that have resulted from use of the Manual? 4. Have you noticed any human health outcomes (e.g. reductions in the number of accidents) that have resulted from this Manual, or are you aware of any potential effects on human health? TOPIC AREA IV: IMPROVING THE MANUAL Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol facilities? 1. What changes would you recommend to the Manual to improve its readability and navigation? 2. What could be changed about the Manual to make it more thorough and comprehensive and to make it more effective as a compliance tool? 3. Are there topics you would like more or new information about in a future Manual? 4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences, or make facilities and other stakeholders aware that an updated Manual is available? 5. Do you have any other suggestions for improving the Manual? TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE Overarching Question: What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? 1. What are other ways you find information about regulations and applicable laws? 2. What are other ways EPA could present the information in the Manual to its targeted audiences? (e.g. workshops, interactive Web site) FINAL QUESTION 1. Do you have any final "closing thoughts" or recommendations regarding the overarching questions, sub questions or the compliance assistance Manual in general? Thank you for participating! November 2009 Page 23 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Appendix E. Interview Guide for State Compliance Staff A NOTE TO INTERVIEWEES A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf If you have any questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206- 447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com. TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? 1. Where did you obtain the Manual? 2. Does your agency refer ethanol facilities and contractors to the Manual? 3. How could distribution of or access to the Manual be improved? TOPIC AREA II: EFFECTIVENESS OF THE MANUAL Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? 1. How readable and comprehendible is the Manual? 2. How comprehensive is the Manual in its scope? 3. Is the contact information provided in the Appendices helpful? How could it be improved? 4. Have you been contacted by an ethanol facility or contractor as a result of the Manual? TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? November 2009 Page 24 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 1. Could compliance with state laws and rules be enhanced using the Manual? How? 2. In your interactions, do facilities seem aware of the applicable Federal regulations? 3. Have you noticed any changes in facility compliance with regulations since the Manual was released? TOPIC AREA IV: IMPROVING THE MANUAL Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol facilities? 1. What changes would you recommend to the Manual to improve its readability and navigation? 2. What could be changed about the Manual to make it more thorough and comprehensive and to make it more effective as a compliance tool? 3. Are there topics you would like more or new information about in a future Manual? 4. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences, or make facilities and other stakeholders aware that an updated Manual is available? 5. Do you have any other suggestions for improving the Manual? TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE Overarching Question: What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? 1. What are other ways EPA could present the information in the Manual to its targeted audiences? (e.g. workshops, interactive Web site) FINAL QUESTION 1. Do you have any final "closing thoughts" or recommendations regarding the overarching questions, sub questions or the compliance assistance Manual in general? Thank you for participating! November 2009 Page 25 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices Appendix F. Interview Guide for EPA Staff A NOTE TO INTERVIEWEES A Ross & Associates evaluator will ask the questions in this guide during your telephone interview. The interview is not intended to be a test of your knowledge of the compliance manual Environmental Laws Applicable to the Construction and Operation for Ethanol Plans. Rather, the evaluation is designed to collect feedback on your thoughts about the Manual's effectiveness in relaying compliance information to ethanol facilities. For the interview, it is not expected that you will have carefully read the entire Manual, but please at a minimum review those parts of the manual that pertain to your area of work. The manual can be accessed at: http://www.epa.gov/region07/priorities/agriculture/ethanol plants manual.pdf If you have any questions, please contact Anna Williams or Amy Wheeless, Ross & Associates by phone at 206- 447-1805 or by email at anna.williams(5)ross-assoc.com or amv.wheeless(5)ross-assoc.com. TOPIC AREA I: OUTREACH AND ACCESS TO MANUAL Overarching Question: How effective are Region 7's efforts to advertise the availability of and distribute the Manual to ethanol facilities and other intended audiences? 1. Does your agency refer ethanol facilities and contractors to the Manual? 2. How could distribution of or access to the Manual be improved? TOPIC AREA II: EFFECTIVENESS OF THE MANUAL Overarching Question: How effective is the Manual as a tool for transferring compliance (and related facility technical support) information to ethanol facilities? 1. How readable and comprehendible is the Manual? 2. How comprehensive is the Manual in its scope? 3. Is the contact information provided in the Appendices helpful? How could it be improved? 4. Have you been contacted by an ethanol facility or contractor as a result of the Manual? TOPIC AREA III: UNDERSTANDING AND BEHAVIOR CHANGE Overarching Question: Has the existence and use of the Manual led to increased understanding and behavior change to support improved compliance? 1. In your interactions, do facilities seem aware of the applicable Federal regulations? November 2009 Page 26 ------- Evaluation of the Effectiveness of an Ethanol Compliance Manual Report Appendices 2. Have you noticed any changes in facility compliance with regulations since the Manual was released? TOPIC AREA IV: IMPROVING THE MANUAL Overarching Question: How could the Manual be improved to provide effective compliance assistance to ethanol facilities? 1. What changes would you recommend to the Manual to improve its readability and navigation? 2. What could be changed about the Manual to make it more thorough and comprehensive and to make it more effective as a compliance tool? 3. What are possible ways that EPA Region 7 could distribute an updated Manual to its audiences, or make facilities and other stakeholders aware that an updated Manual is available? 4. Do you have any other suggestions for improving the Manual? TOPIC AREA V: OTHER COMPLIANCE ASSISTANCE Overarching Question: What approaches other than the Manual would provide effective compliance assistance to the ethanol facilities? 1. What are other ways EPA could present the information in the Manual to its targeted audiences? (e.g. workshops, interactive Web site) FINAL QUESTION 1. Do you have any final "closing thoughts" or recommendations regarding the overarching questions, sub questions or the compliance assistance Manual in general? Thank you for participating! November 2009 Page 27 ------- |