Assessing EPA's Small Discussion Groups Background Information for EPA Staff Introduction Enclosed are three questionnaires for EPA staff to use in generating feedback on the effectiveness of public small discussion groups. The questionnaires are a component of the evaluation section in the Framework for Implementing EPA's Public Involvement Policy. The questionnaires are for people who attend a small discussion group to complete immediately after the sessions, for EPA/contractor staff responsible for coordinating the work of a small discussion group, and for small discussion group participants who receive follow-up information following the small discussion group. The first questionnaire focuses on participant perspectives of the effectiveness of a small discussion group. The second questionnaire focuses on EPA/contractor perspectives about the success of the same small discussion group. The third questionnaire focuses on participant perspectives on the quality of the follow-up information received after the same small discussion group. What is a small discussion group? A small discussion group usually has 6-15 participants. It is often, but not always, composed of like-minded individuals, members of a particular interest group, or a segment of the population (such as seniors). These participants are usually members of the general public, rather than representatives from stakeholder organizations or non-governmental organizations. EPA often conducts a series of small discussion group sessions about a specific topic or Agency activity to solicit input from a range of different like-minded individuals or interest groups, and/or to ensure that the input gathered represents viewpoints from various regions of the country. The EPA staff who hold these sessions should use experienced facilitators who are familiar with participatory methods for generating lively discussions and soliciting input from all the participants. Small discussion groups are most successful when the EPA staff who are intimately familiar with the selected topic lead the discussion, with the assistance of a facilitator and an experienced recorder. The purpose of small discussion group sessions is to generate a discussion, and solicit face-to-face input about a specific EPA-related issue, project, activity, policy or program from various sectors of the general public. Session facilitator(s) encourage all participants to express their opinions about the topic under discussion. How can the small discussion group questionnaires be useful to you? Use of the questionnaires should make the process of gathering feedback for your public involvement activity easier for you to implement directly, without spending contract dollars. The questionnaires provide an easy way to get useful feedback from participants and EPA about the small discussion groups they participated in. This feedback should help you to better understand whether a particular small discussion group worked well, whether there are opportunities for improvement, or whether certain problems need to be addressed. Once you have summarized the data, Agency staff can assess whether they met goals, make modifications, and compare progress over time. When should I use the questionnaires? You can use the first questionnaire (for small discussion group participants) at the end or towards the end of a small discussion group. You can use the second questionnaire (for EPA staff or their contractors who: assisted in planning the session, lead the session, or facilitated the session) after the same small discussion group. You can use the third questionnaire (for small discussion group participants) when sending out follow-up information to all small discussion ------- group participants. It is ultimately up to you, however, to decide when to use them. The key is to use the questionnaires to the extent that they provide you with valuable information and improve your ability to design and implement more effective small discussion groups in the future. Are the questionnaires in compliance with the Paperwork Reduction Act? Yes. The questionnaires were cleared with the Office of Management and Budget (OMB) under ICR #2151.01, OMB Control No: 2010-0039. This clearance will allow EPA to collect information (via the questionnaires) from more than nine non-federal entities. Currently, these questionnaires are undergoing a pre-test. Who designed these questionnaires? The Evaluation Task Group of EPA's Public Involvement Workgroup designed these questionnaires after consulting with EPA staff who are regularly involved in the designing and holding of small discussion groups. What should I do when preparing to administer a questionnaire? See "Checklist for Administering the Questionnaire." What do I do with the questionnaire data once the forms are returned? As soon as you can, record and tabulate the responses you receive to each question on each survey in a format that works for you. Include the date, location, total number of respondents and which survey you used on the form. After each use of the same survey, check the past results so you can track improvement in the respondents' ratings. On an annual basis, EPA must report use of the surveys to the Office of Management and Budget. Just before Thanksgiving, send a summary of your use of all the surveys to: bonner.patricia@epa.gov. Please note all the surveys used, the number of times you administered each, and the total number of respondents to each survey. If you have seen improved ratings, add a note about what you changed and what difference it made in not only the rating, but the activity itself. That way we will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaires are useful to your efforts to improve public involvement activities. If you have suggestions for additional surveys or changes to existing surveys, send those along too. When it becomes available, you may collect, collate and analyze the data from the respondent forms using a pre-formatted Lotus 1-2-3 spreadsheet program. In addition, the spreadsheet program will allow you to summarize data over a period of time, making it easy to generate annual summary reports for your management. More information on how to use the program is available in the "Checklist for Administering the Questionnaire and Using Results" and within the Lotus 1-2-3 spreadsheet program. How can I use the results? The completed questionnaires should provide you with information that can help you design improvements to activities and to your overall involvement process. You can learn what worked well, what participants feel is important and what should change to better accommodate their needs. It will be up to you to decide how much data you need to prompt changes. While it is prudent to be cautious about making changes based upon a small amount of feedback data, you may find a number of the respondents' ideas useful. If you can act on their suggestions, even in a small way, participants will appreciate that you listened. ------- How should I communicate the results? Use the performance and summary reports from the Lotus 1-2-3 spreadsheet program to talk with managers and staff about the effectiveness of the small discussion groups and where changes may be necessary. Communicate overall results and what changes you intend to make to small discussion group participants in the small discussion group summary report or other follow-up materials distributed to session participants. A few brief sentences describing overall what participants thought of the session, their suggestions for improvement, and what you will do to improve the next small discussion group would let participants know that you valued their input. What is EPA's Public Involvement Policy and where can I get more information? EPA's 2003 Public Involvement Policy underscores EPA's commitment to improving its practices and is based on seven steps for effective public involvement: 1. Plan and budget for public involvement activities 2. Identify the interested and affected public 3. Consider providing technical or financial assistance to the public to facilitate involvement 4. Provide information and outreach to the public 5. Conduct public consultation and involvement activities 6. Review and use input, and provide feedback to the public 7. Evaluate public involvement activities The Framework for Implementing EPA's Public Involvement Policy urges development of tools to improve public involvement information sharing, training and evaluation at EPA. This questionnaire and the database program are examples of such tools. Cross-agency work groups developed the Policy and Framework using public comments and ideas gathered through a two-week, Internet-based Dialogue on Public Involvement in EPA Decisions. Copies of the Policy and Framework are available @ http://www.epa.gov/publicinvolvement/public/index.htm. ------- |