MODULE 8 Bringing It All Together: Evaluation and Lessons Learned PARTICIPANTS' GUIDE Published by: U.S. EPA Region 4 Environmental Justice and Children's Health Section Environmental Justice Academy A United States Environmental Protection »* Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Table of Contents Preparation Materials 3 Icon Key 4 Agendas 5 Module 8 Day 1 5 Module 8 Day 2 6 Module 8 Day 1 7 Module 8 Day 1: Welcome Back 8 Module 8 Day 1: Close the Gap Recap: Module 7 8 Module 8 Day 1: Why Evaluate? 8 Module 8 Day 1: A Framework for Conducting Program Evaluations 11 45-Minute Break 13 Module 8 Day 1: Program Evaluation 14 Module 8 Day 1: The Logic of It All 15 Module 8 Day 1: Activity: Building a Logic Model 17 Module 8 Day 1: Closing 19 Module 8 Day 2 20 Module 8 Day 2: Welcome Back 21 Module 8 Day 2: Activity: EJ Academy Alumni Networking Session 21 Module 8 Day 2: Activity: Portfolio Screening 21 Module 8 Day 2: Activity: Using Toxics Release Inventory Data to Identify Facilities for Pollution Prevention Outreach in Underserved Communities 22 Module 8 Day 2: Activity: Exploring EPA's Data Mapping Tools 22 Module 8 Day 2: Activity: Working Lunch, Graduation Plans, and Portfolio Review Wrap-Up 23 Module 8 Day 2: Activity: Participation with Local Environmental Organization 23 Homework Assignment 23 Module 8 Day 2: Closing 24 Appendices 25 Appendix A: Additional Resources 26 Appendix B: Six Blind Men and an Elephant Story 27 Appendix C: Logic Model Template 28 Appendix D: Full Portfolio Review 29 Appendix E: Poster Presentation Template 43 Appendix F: Collaborative Problem-Solving Element 7 44 United States Environmental Protection kl a m Agency ------- Bringing It All Together: Evaluation and Lessons Learned Additional Notes: Module 8 Participants' Guide United States Environmental Protection kl a m Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Preparation Materials Participant Icon Key and Agendas United States Environmental Protection kl a m Agency 3 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Icon Key Icon Title Meaning 0 Instructions to the facilitator This icon means there are specific instructions for the facilitator. This content should not be read aloud. m 1 Background for the facilitator This icon indicates that there is background information the facilitator should be aware of when covering this topic. This information is presented in the facilitator's guide. Facilitator says This icon means the facilitator should read the content nearly verbatim, interjecting thoughts when appropriate. 0 Key point This icon indicates an important point that the facilitator should communicate to the audience in their own words. I 1 1 Questions This icon means the facilitator should allow time for the audience to ask questions or the facilitator should ask the audience questions. D Activity This icon indicates an activity that the facilitator should explain. The activity can be either a group or individual activity. . Assessment This icon indicates the facilitator will conduct an assessment. Multimedia This icon indicates the facilitator will have students watch a video. | O Computer This icon indicates the facilitator will have students visit an online resource. Handouts or resource materials This icon indicates the facilitator will give students a handout or other resource material. - Homework This icon indicates a homework assignment. c Transition This icon indicates the facilitator will provide a transition from one topic to another. United States Environmental Protection kl a m Agency 4 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Aqendas Module 8 Day 1 Bringing It All Together: Evaluation and Lessons Learned Date: Time: Location: Contact: Time Day 1 Facilitator Close the Gap Recap: Module 7 Sign in Overview of module 7 Short homework review Questions and answers Lesson 1Why Evaluate? Importance of evaluation Two overarching types of evaluation: process and outcome Lesson 2A Framework for Conducting Program Evaluations Review the six components of the evaluation framework Break Lesson 3Program Evaluation Define evaluation stakeholders Lesson 4The Logic of It All What is a logic model? Logic model components What are the logic model pathways? How does a logic model help your evaluation? ActivityBuilding a Logic Model Closing Remarks Short introduction to day 2 Thanks and closing of session United States Environmental Protection kl a m Agency 5 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 2 Bringing It All Together: Evaluation and Lessons Learned Date: Time: Location: Contact: Time Day 2 Facilitator Welcome/Check-In EJ Academy Alumni Networking Session EJ Academy alumni Current EJ Academy students Portfolio Screening Review action plan elements Using Toxics Release Inventory Data to Identify Facilities for Pollution Prevention Outreach in Underserved Communities Break Exploring Environmental Health Issues with Office of Research and Development (ORD) Community Decision Support Tools Recent advancements in EJScreen Working Lunch Portfolio review wrap-up Discuss graduation activities United States Environmental Protection kl a m Agency 6 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 1 Bringing It All Together: Evaluation and Lessons Learned United States Environmental Protection kl a m Agency 7 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 1: Welcome Back Objective: Welcome back to the EJ Academy. Sign In Sign the class roster and ensure that the points you have earned are annotated on your participant points matrix. Module 8 Day 1: Close the Gap Recap: Module 7 Objective: Discuss the purpose of your piranha bowl presentations and the feedback you received from the judges. B Module 7 Overview Piranha Bowl The "piranha bowl" was a small-scale simulation of the business funding-related reality show, "Shark Tank." Like the show, you were required to persuade potential stakeholders to invest in your plan for your community. This exercise was designed to prepare you to pitch your action plan, program, or project to actual stakeholders or potential investors during the graduation and networking session. Discussion Points What stood out to you? What concepts excited you or resonated with you? How can you use this information? Module 8 Day 1: Why Evaluate? Objective: Define and understand program evaluation. Defining Program Evaluation Program evaluation is a systematic, objective process for determining the success of a policy or program. It addresses questions about whether and to what extent the program is achieving its goals and objectives. United States Environmental Protection kl a m Agency 8 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Resource-providing organizations, such as EPA, are typically challenged with prioritizing competing demands for investments. These organizations work on a wide variety of challenges, such as housing, health care, education, the environment, labor, economic development, crime, public safety, and comprehensive community change. stronger and have clearer goals, strategies, actions, and measures of success. Program evaluation helps to clarify the project's underlying assumptions and causal relationships. In addition, evaluation can help organizations proactively identify opportunities to adjust as early as possible. To be effective, evaluation needs to be embedded within the culture of the community organization and its stakeholders. Program evaluation has many benefits: Measures program effectiveness. Improves the program by identifying strengths and opportunities for advancement. Informs stakeholders and decision-makers of program success. Demonstrates what the program is accomplishing. Provides support for decision-making and strategic planning. Improves program sustainability. Helps gain resources and funding from potential program benefactors. Helps the program adapt to change. Program evaluation also helps projects meet two different but interrelated goals: Addressing the program development needs of community-based efforts. Addressing the accountability needs of organizations that provide resources (i.e., financial and technical assistance) for community-based efforts. Why Evaluate Programs? To gain insight about a program and its operationsto see where it is going and where it came from, and to find out what works and what doesn't. To improve efficiency and modify practices to increase success. To assess effectsto see how well the program meets objectives and goals and benefits the community, and to provide evidence of effectiveness. To build capacity by increasing funding, enhancing skills, and strengthening accountability. Importance of Program Evaluation Projects that incorporate evaluation from the very beginning are usually United States Environmental Protection kl a m Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide jq Two Overarching Types of Evaluation: Process and Outcome Evaluation Getting intermittent, honest feedback from partners, members, and other stakeholders can help an organization adjust the direction of the project and refine specific aspects. Evaluation helps organizations continuously improve during projects, which is more advantageous than waiting until the end. Process Evaluation A process evaluation looks at the development and implementation of a program. It establishes whether you've hit quantifiable targets and implemented strategies as planned. It's typically done at the end of a project and looks at the program from start to finish, assessing cause and effect relationships between the program components and outcomes. This type of evaluation can be very useful in determining whether a program should be continued, expanded upon, refined, or eliminated. Questions that may be asked during a process evaluation include: Has the project reached the target group? Are all project activities reaching all parts of the target group? What aspects of the project are participants and other key stakeholders satisfied with? What activities are being implemented as intended? What additional activities can be implemented? What, if any, changes have been made to the planned activities? Are all materials and presentations suitable for the target audience? Outcome Evaluation Outcome evaluation measures the effectiveness of a program in producing change. It examines the short-term, intermediate, and long-term impact of a program or project. This type of evaluation is generally used to measure how successfully the program has achieved its goals. Although outcome evaluation measures the main goal of the program, it can also be used to assess program objectives overtime. For example, an organization's process evaluation might confirm that 200 people have completed its skills training program. An outcome evaluation would tell the organization how many of those people demonstrated increased confidence, changed behaviors, or found jobs because of their new skills. Questions that may be asked during an outcome evaluation include: Has the overall program goal been achieved? What, if any, factors outside the program have contributed to the desired change? United States Environmental Protection kl a m Agency 10 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide What, if any, unexpected changes occurred because of the program? Discussion Points What types of evaluations have you completed in the past? Which evaluations worked best? What steps would you take to plan and implement your own evaluation? Evaluation in the Real World Discuss program evaluation. Review real world examples that demystify evaluation. Evaluation What It Measures/Assesses Restaurant ratings Sanitation, food temperatures, food quality Lab results Biological markers to reference range Consumer reports or other online reviews Quality and value of items Polling numbers Trends in public opinion Car gas gauge Amount of gas available School report cards School performance Module 8 Day 1: A Framework for Conducting Program Evaluations Objectives: Learn key evaluation terms. Learn how to conduct a program evaluation by using a program evaluation framework. jQ Evaluation Approaches There are many theories and approaches to evaluation. The specifics of each approach are not important, just that there are different ways to approach evaluation. In this module, you will learn six steps to evaluation that are based on a useful framework. United States Environmental Protection kl a m Agency 11 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Steps Engage Stakeholders Ensure use and share lessons learned Describe the program Justify conclusions Standards Utility Feasibility Propriety Accuracy Focus the evaluation design Gather credible evidence This evaluation framework was originally developed by the Centers for Disease Control and Prevention (CDC).1 Reference to specific commercial products, manufacturers, companies, or trademarks does not constitute its endorsement or recommendation by the U.S. Government, Department of Health and Human Services, or Centers for Disease Control and Prevention. Evaluation Framework Components The six steps of program evaluation are shown in a circle representing the circular nature of the evaluation process. The steps are interdependent, and they may be conducted in a nonlinear order. Often, more than one step is ongoing at any time. The framework provides the flexibility to go backwards or forwards in the process depending upon an organization's needs. Although the steps can be conducted in a different order or simultaneously, there is a reason for the way the steps have been sequencedthe earlier steps provide the foundation for the later ones. Each step plays a crucial role in the evaluation process. 1 Centers for Disease Control and Prevention (CDC), (n.d.). A Framework for Program Evaluation. http://www.cdc.gov/eval/framework/index.htm United States Environmental Protection kl a m Agency 12 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide The circle also represents that the program evaluation process is iterative. As program evaluation is conducted, the information or findings from the evaluation are used to improve the program. The process repeats itself in a cycle to improve the program and its effectiveness in achieving its objectives. Engage stakeholders. Including multiple perspectives will help you develop a robust and useful program evaluation that allows for more inclusivity. Describe the program. To conduct a program evaluation, you and your stakeholders need to be clear about what program will be evaluated and in what context. Evaluators should create a detailed outline of each component and describe its effectiveness. Focus the evaluation design. Identify what information you want to gain from the evaluation given the resources available. Gather credible evidence. Make sure you are gathering information that stakeholders will consider relevant to answering the key evaluation question. Consider what needs to be collected (i.e., indicators), where the information will come from (i.e., data sources), how the information is collected (i.e., data collection methods), and the desired quality and quantity of information. Justify conclusions. In this step, you will link the evidence gathered and compare it with the agreed-upon values or standards set by your stakeholders. Ensure use and share lessons learned. Focus on preparing the results to share and gain feedback. The value of the evaluation results depends on their use. 45-Minute Break Objective: Take a break before the next section. Take a 45-minute break. Overview of Steps United States Environmental Protection kl a m Agency 13 ------- Bringing It All Together: Mod u le 8 Evaluation and Lessons Learned Participants' Guide Module 8 Day 1: Program Evaluation Objective: Learn how to engage stakeholders using the evaluation process. Define Evaluation Stakeholders Stakeholders are people or organizations invested in the program, interested in the results of the evaluation, and/or with a stake in what will be done with the results of the evaluation. There are three categories of stakeholders: Those involved in program operations, such as management, program staff, partners, funding agencies, and coalition members. Those served or affected by the program, such as patients or clients, advocacy groups, community members, and elected officials. Those who are intended users of the evaluation findings or are able to make decisions about the program, such as partners, funding agencies, coalition members, and the general public. o o Discuss: Audience Poll Who are some of the evaluation stakeholders in your community? Who cares about the environment? Possible answers: residents, students, business owners, schools, public officials, homeowners' organizations, and churches. Who is in a position of authority or power in your community? - Possible answers: city council members, mayors, principals, church leaders, the president of the homeowners' association, and the local department of public health or code enforcement. Why is stakeholder involvement critical to the success of your evaluation? Program evaluation created in a vacuum (i.e., without stakeholder input) may not be useful. Stakeholder involvement can help organizations identify data sources, ensure data collection tools are relevant to the community, and leverage resources. Stakeholders can help interpret results and determine what the key findings are. What roles may stakeholders have in the program evaluation? For example, stakeholders might: o Serve on an evaluation advisory group. o o o United States Environmental Protection kl a m Agency 14 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide o Help define key evaluation questions and the overall evaluation plan, o Identify resources for the evaluation in terms of time, funds, and access to data. o Pilot test surveys or review data collection tools, o Help interpret evaluation results that may be confusing, o Use the evaluation results to make environmental changes. Discuss How can your organization get community input? What are some things to consider when getting input from the community? What works in getting this community involved? Will trust have an impact on getting community input? What can be done to build trust and get community input? Module 8 Day 1: The Logic of It All Objective: Complete a program evaluation using the logic model method. Wi Before conducting an evaluation, you need to understand what will be I evaluated. Having a clear description of your program makes sure everyone involved has a shared understanding of what will be done and a common view of what the program is trying to achieve. A logic model is a useful tool for describing a program. The six blind men and an elephant story can be found in Appendix B. n V V Story Summary In a parable, six blind men were challenged to accurately describe an elephant. To add to the challenge, each man was taken to the elephant alone and allowed to touch only one area of the animal. When they were asked to describe the elephant, they each had a completely different perspective. This retelling of a traditional story from India illustrates how different people can have distinctly different perceptions of the same thing. What Is a Logic Model? A logic model is a systematic and visual way to present the perceived relationships among the resources you need to operate the program, the United States Environmental Protection kl a m Agency 15 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide activities you plan to do, and the results you hope to achieve. It also includes some context of the environment in which you are trying to implement your program or effort. Since logic models are visual, you need to think systematically to best describe your program. The visual representation points out areas of strength or opportunities for growth and can be adjusted as your program develops. It helps to clarify assumptions with your stakeholders and program staff and to create a shared understanding of the program's goals, activities, and expected outcomes. a Logic Model Components Inputs include the human, financial, organizational, and community resources a program has available to direct toward the work. Activities are what the program does with the resources. Activities are the processes, tools, events, technology, and actions that are an intentional part of the program's implementation. These interventions are used to bring about the program's intended results. Outputs are the direct products of program activities and may include the types and intended targets of services the program intends to deliver. Outcomes (e.g., short-term, intermediate, and long-term) are the specific changes in program participants' behavior, knowledge, skills, status, and level of functioning. Short-term outcomes should be attainable within one to three years, while long-term outcomes should be achievable within a four to six year time frame. The logical progression from short-term to long-term outcomes should be reflected in impacts occurring within about seven to 10 years. What Are the Logic Model Pathways? The logic model pathways are how the inputs, activities, outputs, and outcomes connect with each other across the logic model. Do the links make logical sense? A common pitfall in logic model development is not incorporating activities that result in the outcomes expected. How Does a Logic Model Help Your Evaluation? A logic model helps describe the core elements of your program and the desired outcomes of your efforts. Your evaluation efforts may focus on a particular box of the logic model or the relationship between the boxes. United States Environmental Protection kl a m Agency 16 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Program Action - Logic Model Situation Needs and assets Symptoms versus problems Stakeholder engagement Priorities Consider: Mission Vision Values Mandates Resources Local dynamics ( Collaborators Competitors Intended outcomes Inputs Outputs Activities Participation What we invest Staff Volunteers Time Money Research base Materials Equipment Technology Partners What we do Conduct workshops, meetings Deliver services Develop products, curriculum, resources Train Provide counseling Assess Facilitate Partner Work with media Who we reach Participants Clients Agencies Decision- makers Customers Satisfaction Assumptions Outcomes - Impact Short Term Medium Term Long Term 9 What the What the What the short term medium term ultimate results are results are impact(s) is Learning Action Conditions Awareness Behavior Social Knowledge Practice Economic Attitudes Decision- Civic Skills making Environmental Opinions Policies Aspirations Social Action Motivations External Factors Evaluation Focus - Collect Data - Analyze and Interpret - Report Module 8 Day 1: Activity: Building a Logic Model Objective: Learn how the logic model helps organizations achieve desired outcomes and identify a program's strengths, weaknesses, and opportunities to improve. What do we hope to accomplish as a result of our program? What do we need to do to achieve these outcomes? What resources do we need to implement our program and achieve the desired results? What is going on in the community that may affect the quality and impact of our program? JV United States Environmental Protection IhI a m Agency 17 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Logic Model Worksheet Inputs Strategies Outputs Outcomes All the resources necessary for supporting a program. Activities, services, or interventions for the program's targeted audience. Process measures linked to strategies and program implementation. The anticipated effects of your program. Examples: Financial resources Program staff Examples: Training Outreach Examples: Number of people trained Number of outreach materials created Examples: Increased skills Improved performance Your Inputs Your Strategies Your Outputs Your Outcomes Contextual Factors (i.e., Factors Beyond Your Control) Finalizing the Logic Model Place your responses in the appropriate sections of the logic model diagram. Draw arrows to depict relationships. Review the model with other groups and revise as needed. United States Environmental Protection kl a m Agency 18 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 1: Closing Objective: Close out day 1 of module 8 Summary of Module 8 Day 1 By the end of module 8 day 1, you should be able to: Define and understand program evaluation. Conduct a program evaluation using a program evaluation framework. Engage stakeholders using the evaluation process. Complete a program evaluation using the logic model method. jq Short Introduction to Day 2 The facilitator has three choices for how to best implement day 2. They might also implement a combination of these three choices. On day 2, you might: 1. Attend an environmental justice-related conference. 2. Learn to use decision-making tools provided by EPA's Office of Research and Development and engage with a local environmental community organization. 3. Learn how to begin citizen science projects in your community and engage with community-based participatory research practices. n Thanks and Closing of Session Thank you for devoting your time to attending the EJ Academy. We hope that once you complete the program, each of you will return to your community to impart newly developed skills to other members, spreading the influence of the EJ Academy and empowering them from within. United States Environmental Protection kl a m Agency 19 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Dav 2 Bringing It All Together: Evaluation and Lessons Learned vvEPA United States Environmental Protection Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 2: Welcome Back ~ Once you sign in, you will have an opportunity to meet EJ Academy alumni who completed the program in previous years and are attending this session to help review and improve your portfolios. Module 8 Day 2: Activity: EJ Academy Alumni Networking Session Objective: Meet and network with EJ Academy alumni who will help you improve your portfolios. c Module 8 Day 2: Activity: Portfolio Screening Objective: Improve your portfolio. Activity: Review the current EJ Academy participants' portfolios. The facilitator will give you the "Full Portfolio Review" handout located in Appendix D. The facilitator will pair you with either EJ Academy alumni or EJ Academy staff to execute the portfolio review. You can also use this time to review the portfolios of participants from previous cohorts. This will help you better understand of what is expected in regards to portfolio completion, style, grammar, and presentation. Remember that these are merely examples and that everyone will produce a ( portfolio that is equivalent^ unique and important. Remember that you will be turning in new drafts of previously completed assignments, not a whole new assignment. Feel free to ask questions to clarify the portfolio criteria. United States Environmental Protection kl a m Agency 21 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Module 8 Day 2: Activity: Using Toxics Release Inventory Data to Identify Facilities for Pollution Prevention Outreach in Underserved Communities Objective: Learn about an online database to engage industry. ~ This lesson will cover the Toxics Release Inventory (TRI) as a means to encourage community involvement. The database is available at the TRI Program website: https://www.epa.aov/toxics-release-inventorv-tri-proaram The TRI is a resource for learning about toxic chemical releases and pollution prevention activities reported by industrial and federal facilities. TRI data helps communities, government agencies, companies, and others make informed decisions. 6^ Successfully motivating a community to actively bring about environmental change requires creating awareness of what needs to change. Therefore, the starting point for any community leader or group is proving the need for change. No one will ever change what they can't measure. Finding the right tools to measure change is key in creating community involvement. The TRI is one of those tools and is available free of charge. Another tool to supplement the TRI database is EPA's Environmental Compliance History Online (ECHO) tool, which outlines how facilities in your community are doing their part to comply with environmental laws (see https://echo.epa.gov/). Not only do these tools help you measure the current environmental condition of a community, they can also show you a community or facility's progress through the years. The facilitator will now discuss another online tool created by EPA's Office of Research and Development (ORD). Module 8 Day 2: Activity: Exploring EPA's Data Mapping Tools Objective: Learn about EPA-created tools that can help you understand your community's environmental health issues. The facilitator will present the latest version of EJScreen, as well as other EPA tools that are relevant to the class. EJScreen 2.0 is an environmental justice screening and mapping tool based on nationally consistent data and an approach that combines environmental and United States Environmental Protection kl a m Agency 22 ------- Bringing It All Together: Mod u le 8 Evaluation and Lessons Learned Participants' Guide demographic indicators in maps and reports. The tool is available at https://www.epa.gov/eiscreen. Module 8 Day 2: Activity: Working Lunch, Graduation Plans, and Portfolio Review Wrap-Up Objectives: Finish portfolio reviews and discuss the graduation ceremony plans. Activity: Working break to discuss graduation and portfolio review wrap-up. Wrap up final conversations or suggestions regarding the current draft of your portfolios. The portfolio will help you prepare to apply for funding or engage key stakeholders when you are ready. Please remember that the graduation will include showing your portfolio. You are encouraged to invite family, friends, community members, and stakeholders that you would like to engage to continue developing partnerships. Module 8 Day 2: Activity: Participation with Local Environmental Organization Objective: Engage with or put in sweat equity with a local environmental organization. Learn more about citizen science and how to engage with monitoring the environmental parameters in your community. Activity (Optional): Participate in a "green" volunteering opportunity. Homework Assignment Objective: Discuss the poster presentation. Poster Presentation Instructions In addition to preparing your portfolio, you will be required to display a poster presentation showcasing your project for your colleagues, project partners, and other guests during graduation weekend. Your poster should include details on your community background, mission and vision, project goals, action steps, time frames, budget, financing, and acknowledgements to those who helped with your project. The template, available in Appendix J in module 5, is an example of how the poster can be organized. You can use a traditional poster board to convey the details of your project, but you can also use other methods, such as a PowerPoint presentation or photo collage. United States Environmental Protection kl a m Agency 23 ------- Bringing It All Together: Mod u le 8 Evaluation and Lessons Learned Participants' Guide Be prepared to set up your poster/display at least 1.5 hours before the graduation ceremony. Module 8 Day 2: Closing Objective: Close out day 2 of module 8. Thanks and Closing of Session Thank you for devoting your time to attending the EJ Academy. We hope that once you complete the program, each of you will return to your community to impart newly developed skills to other members, spreading the influence of the EJ Academy and empowering them from within. United States Environmental Protection kl a m Agency 24 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendices United States Environmental Protection kl a m Agency 25 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix A: Additional Resources CDC, Introduction to Program Evaluation for Public Health Programs: A Self- Study Guide: https://www.cdc.aov/evaluation/auide/index.htm CDC, Introduction to Program Evaluation for Public Health Programs: A Self- Study Guide, Glossary: https://www.cdc.aov/evaluation/auide/alossarv/index.htm W.K. Kellogg Foundation, Logic Model Development Guide: https://wkkf.issuelab.ora/resource/loaic-model-development-guide.html Urban Institute Series on Outcome Management for Nonprofit Organizations: o "Key Steps in Outcome Management" by Linda M. Lampkin and Harry P. Hatry: http://www.urban.ora/publications/310776.html o "Finding Out What Happened to Former Clients" by Ritu Nayyar-Stone and Harry P. Hatry: http://www.urban.org/publications/310815.html o "Developing Community-wide Outcome Indicators for Specific Services" by Harry P. Hatry, Jake Cowan, Ken Weiner, and Linda M. Lampkin: http://www.urban.org/publications/310813.html o "Surveying Clients About Outcomes" by Martin D. Abravanel: http://www.urban.org/publications/310840.html o "Analyzing Outcome Information" by Harry P. Hatry, Jake Cowan, and Michael Hendricks: http://www.urban.org/publications/310973.html o "Using Outcome Information" by Elaine Morley and Linda M. Lampkin: http://www.urban.org/publications/311040.html Performance Measurement: Getting Results (second edition) by Harry Hatry: https://rowman.com/ISBN/9780877667346/Performance-Measurement-Getting- Results-Second-Edition "Measuring Program Outcomes: A Practical Approach" by the United Way of America: https://digitalcommons.unomaha.edu/cg i/viewcontent.cgi?article=1047&context= slceeval Ten Steps to a Results-Based Monitoring and Evaluation System by Jody Zall Kusek and Ray C. Rist: https://openknowledge.worldbank.Org/handle/10986/14926 "Focusing on Program Outcomes: A Guide for United Ways" by United Way Worldwide: https://www.unitedwavnems.org/wp- content/uploads/2018/02/Qutcomes Guide Final-08.28.09.pdf Benchmarking for Nonprofits: How to Measure, Manage, and Improve Performance by Jason Saul My-Peer Toolkit, "Types of Evaluation": http://mypeer.org.au/monitoring- evaluation/types-of-evaluation/ United States Environmental Protection kl a m Agency 26 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix B: Six Blind Men and an Elephant Story Once upon a time, there lived six blind men in a village. One day the villagers told them, "Hey, there is an elephant in the village today." They had no idea what an elephant was. They decided, "even though we would not be able to see it, let us go and feel it anyway." All of them went to where the elephant was. Every one of them touched the elephant. "Hey, the elephant is a tree," said the first man, who touched the leg. "Oh, no! It is like a rope," said the second man, who touched the tail. "Oh, no! It is like a thick snake," said the third man, who touched the trunk. "It is like a big hand fan," said the fourth man, who touched the ear. "It is like a huge wall," said the fifth man, who touched the belly. "It is like a spear," said the sixth man, who touched the tusk. The men began to argue about the elephant, and every one of them insisted that he was right. It looked like they were getting agitated. A wise man was passing by and saw this. He stopped and asked them, "what is the matter?" The men said, "we cannot agree about what the elephant is like." Each one of them said what he thought the elephant was like. The wise man calmly explained to them, "all of you are right. Every one of you is telling it differently because each one of you touched a different part of the elephant. The elephant actually has all the features you described." "Oh!" everyone said. There was no more fight. They felt happy that they were all right. The moral of the story is that there may be some truth to what everyone says. Sometimes we can see that truth and sometimes not, because they may have a perspective we don't agree with. So, rather than arguing like the blind men, we should accept other people's perspectives to avoid arguments. JV United States Environmental Protection IhI a m Agency 27 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix C: Logic Model Template Logic Model Development Program Implementation Template Resources Activities Outputs Short- and Long-Term Outcomes Impact In order to accomplish our activities, we will need the following: To address our problem, we will accomplish the following activities: l/l/e expect that once accomplished, these activities will produce the following evidence of service delivery: l/l/e expect that if accomplished, these activities will lead to the following changes in one to three years, then in four to six years: l/l/e expect that if accomplished, these activities will lead to the following changes in seven to 10 years: A United States Environmental Protection ^1 l m Agency 28 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix D: Full Portfolio Review EJ Academy Participant: [Name] Portfolio Requirements Completed Notes 1. Community Executive Summary a. Piranha bowl summary b. Dream statement c. S.M.A.R.T. action plan 2. Community Description a. Organization and/or community name b. Organizational chart c. Community background and historical information (see requirements i-iv) i. Challenges #1, #2, and #3 ii. History and background (i.e., boundaries, geography) iii. Demographic information (for community residents and organizational members) iv. Community and natural resources 3. Stakeholder Partnerships and Resources Chart Stakeholder partnerships and resources chart 4. Budget Budget 5. Financial Resources and Assets Financial resources and assets United States Environmental Protection kl a m Agency 29 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide 1a. Community Executive Summary Piranha Bowl Presentation Summary Name Name of organization/affiliation Title of program/project Required Elements Mission/vision Project plan Importance to potential partner (Why should the piranha be attracted to supporting your program/project?) Budget/financial investment United States Environmental Protection kl a m Agency 30 ------- Bringing It All Together: Mod u le 8 Evaluation and Lessons Learned Participants' Guide 1b. Dream Statement Include either your short, logo-style dream statement or your longer, more detailed dream statement. United States Environmental Protection kl a m Agency 31 ------- Bringing It All Together: Mod u le 8 Evaluation and Lessons Learned Participants' Guide 1c. S.M.A.R.T. Action Plan Goal: Use chart below. You should have completed at least one section per module, based on the module topic. S.M.A.R.T Actions Actions should contribute to success of a goal and be S.M.A.R.T.: Specific, measurable, achievable, relevant:, and time-bound Why Is This Important? Supporting Cast Who else can help? Costs and Resources Funding sources? Human resources? Time Frame Short (1-2 years), medium (3-5 years), long (>5 years) Lead Role Who is the primary responsible organization or person ? How Will You Measu re Your Success? Specific action: Specific action: Specific action: Specific action: A United States Environmental Protection ^1 l m Agency 32 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide 2. Community Description 2a. Organization and/or Community Name Name of Organization/Community: 2b. Organizational Chart The instructions for creating an organizational chart from homework 3 are below. Use the organizational chart instructions to prepare an organizational chart that illustrates your core team, including the leaders and key people in your organization and/or community. In each box, list the person's name, their title (if they don't have an official title, list their function/role), and their connection to other people within the organization/community. The instructions can be found in Appendix C in module 3. 2c(i). Community Background and Historical Information Include key parts from module 1 for 2c(ii-v) (shown below). 2c(ii). Challenges #1, #2, and #3 Identifying the Challenges: Discover the Truth Take on the Challenge: Properly Assessing and Identifying the Challenge Module 1 prompted a discussion of the issues within your neighborhood. The first step to solving the problem is not seeing it as a roadblock, but as a challenge that will act as a platform for building up your community. Second, you must properly identify the challenge and ensure that others in your community agree that it is the major 1e a strong foundation for your platfm^ ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Fill out the information below using the activity where you identified your community's challenges (module 1 day 2: We Want Justice activity). Compare this information to the responses you receive from individuals at a community meeting. If you are unable to attend a community meeting, conduct interviews with several community members and ask what they think the major challenges of the community are. Then, use what you have learned and take your perspective and the community's perspective to compile three major challenges within your community. List the challenges from greatest to least importance: Challenge #1: Challenge #2: Challenge #3: United States Environmental Protection kl a m Agency 34 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide 2c(iii). History and Background Define Your Community: On the Road to Discovery Community Assessment Every community has a history that shapes the present state of its environment. It is important for communities to use their past success and mistakes to determine where they would like to be in the future. Research the history of your community at least for the past 25-50 years. Search on the internet and interview community members who have witnessed your community changing. Then, write one or two paragraphs that discuss the history of your community. Include items such as community transitions; important events; and changes in leadership, politics, social climate, or the environment. Find Your Edge: Boundaries/Geography of Your Community It is important to know the boundaries of your community so that you can focus your goals and identify others who can assist within your community. This exercise will help you understand the benefits and challenges that directly affect the people within those boundaries. Fill out the information below using the resource links in module 1, Appendix M. For further instructions, see the example homework assignment, which has more supplementary material. Mention this topic in the community organization's meeting to get feedback on the social boundaries within your community. City, state: What are the geographical boundaries for your community? What are the neighboring communities, based on the boundaries of your community? United States Environmental Protection kl a m Agency 35 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Approximate size, in numbers, of your community's population based on the boundaries determined: Physical boundaries (e.g., rivers, creeks, climate, landforms) in your community: Social boundaries (e.g., transportation, values, knowledge, lack of leadership, lack of collective organization) that limit the communication, collaboration, and the improvement of your community. Be very specific to your community. 2c(iv). Community Assessment The Faces of Change: Demographic Information Many communities recognize that society is constantly putting a face to their community, to who they are. It is time that the people within the community define their community and define the faces of change. Fill out the blanks using census data (use approximate percentages unless otherwise indicated) to determine the demographics (e.g., race, ethnicity, gender, age, income) of your community. If you are within an organization or have a centralized community body, take your research a step further and assess the demographics of those who attend meetings, identifying the demographics of members and group leaders. United States Environmental Protection kl a m Agency 36 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide United States Environmental Protection kl a m Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Residents Group Members Group Leaders Race/Ethnicity Black or African American Asian White Hispanic or Latino American Indian and Alaska Native Native Hawaiian and other Pacific Islander Other (two or more races) Gender Male Female Age Median (Number Value) Educational Attainment High school graduate or above Bachelor's degree or higher Median Household Income (Number Value) Poverty Language (Optional): Speak only English Speak a language other than English Origin Born in the U.S., including territories (native) Born outside the U.S. Cforeian bornl United States Environmental Protection kl a m Agency 38 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide United States Environmental Protection kl a m Agency ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide 2c(v). Community and Natural Resources Using the table below, list only the resources within your community (e.g., individuals, institutions, industries). Pulling From Your Resources: Discovering What Is in Front of You Resources are vital when it comes to accomplishing projects and improving your environment. Many people think of resources as finances or money, but resources can also be people, opportunities, and knowledge. Identify people, organizations, or programs that can act as resources to furthering the vision of your community. Fill out the table below with possible resources you could collaborate with on a project or event. List specific resources that your organization/community is most likely to collaborate with, as well as resources that your community is less likely to collaborate with or would be harder to partner with. If you are an organization or a centralized community group, you should discuss this at a meeting. If not, brainstorm individuals or organizations in your community that have a similar vision or can contribute to improving your community. (These should be possible collaborations, and no formal agreements should be made.) Note: Other resources can refer to assets such as human capital (skills, education, health); cultural capital (traditions, language); natural capital (air, soil, water); and social capital (leadership, groups, networks). Mention these assets on a separate sheet of paper. Local Resources for Furthering Your Community/Organization's Vision Individuals Community groups, services, and organizations (nonprofits, libraries, churches, recreation centers) Institutions (schools, governmental) Industries (restaurants, small businesses, companies) United States Environmental Protection kl a m Agency 40 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide United States Environmental Protection kl a m Agency 41 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide 3. Stakeholder Partnerships and Resources Chart Stakeholder or Partner Sector What Is Their Role In This Effort? Contact Information 4. Budget No specific template required. 5. Financial Resources and Assets No specific template required. United States Environmental Protection kl a m Agency 42 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix E: Poster Presentation Template ^¦DS7x ' A vu Mj ENVIRONMENTAL JUSTICE ACADEMY TITLE (Subtitle) Author Community Background Goal Time Frames Insert picture Mission and Vision Insert picture Action Steps Insert Timeline Financing Insert picture Acknowledgements United States Environmental Protection !¦! m m. Agency 43 ------- Bringing It All Together: Evaluation and Lessons Learned Module 8 Participants' Guide Appendix F: Collaborative Problem-Solving Element 7 Collaborative Problem-Solving Element 7: Evaluation, Lessons Learned, and Replication of Best Practices Objectives: Periodically evaluate progress. Use lessons learned as the project proceeds. Clearly describe measures of success. Document and share information on successes. Techniques used in the evaluation process: Use a logic model to develop a template for the project plan. Define clear measures of success for project outcomes. Develop mechanisms to integrate the lessons into future efforts as new issues and challenges emerge. Share, publish, and disseminate lessons learned and best practices. United States Environmental Protection kl a m Agency 44 ------- |