RE-Powering America's Land Initiative Evaluation Scoping Assessment Fact Sheet April 2015 Introduction Located within OSWER's Center for Program Analysis, the RE-Powering America's Land Initiative aims to promote renewable energy (RE) development on current and formerly contaminated lands, landfills, and mine sites (CLs), when such development is aligned with the community's vision for the site. The Initiative achieves these ends through tailored redevelopment tools, sharing of best practices and success stories, outreach and partnerships, and site-specific technical support from EPA and the Department of Energy's National Renewable Energy Laboratory (NREL). In October 2014, the RE-Powering America's Land Initiative released the final version of its Action Plan 2.0, which articulates the goals and objectives of the Initiative and the activities it expect to pursue over the next two years. In that plan, the Initiative noted its intention to embark on a staged evaluation of its activities, stating that such an evaluation would articulate outcomes, examine the mechanisms used by the Initiative, and explore metrics to measure effort and impact. As a first step, OSWER conducted an evaluation scoping assessment for the RE-Powering Initiative, which has now been completed. The objective of this evaluation scoping assessment was to assess the readiness of the RE-Powering Initiative for an outcome evaluation, focusing on the Initiative's effectiveness and overall contributions to the siting of RE systems on CLs. The project aimed to identify existing data that could be used to conduct an outcome evaluation, and identify any new data that would be required to assess the program's outcomes. The results are intended to inform EPA management decisions on whether and how to proceed with an evaluation of the program. The project team included representatives from: OSWER Center for Program Analysis, RE-Powering managers and staff, and the contractor Industrial Economics (IEc). Evaluation Questions IEc and EPA staff developed a list of 17 potential evaluation questions and then narrowed this list based on initial findings to the following priority questions: How effective have the program's resources, tools, and knowledge products been in raising awareness and encouraging consideration of RE projects on CLs? How effective has the program's site-specific assistance been in raising awareness and encouraging consideration of RE projects on CLs? How effective are the program's resources, tools, and knowledge products; site-specific assistance; and partnerships in addressing barriers to developing RE projects on CLs? How successful is the program in converting leads into interim milestones? What opportunities exist for strengthening the program's lead conversion rate? To what extent have RE projects influenced by the RE-Powering Initiative avoided development on undisturbed lands? What are the avoided or reduced development costs of RE projects on CLs rather than undisturbed lands? Evaluation Scoping Methods The evaluation team conducted the following activities: Reviewed documents and web links to become familiar with the program's history, goals, and activities; Developed a program logic model for the RE-Powering Initiative; Investigated the extent to which each potential evaluation question can be addressed using existing data, and identified potential data or methodological limitations; Identified areas in which the program could potentially enhance its data collection to support future evaluation activities; and Developed a memo with methods and data sources that could be used to answer the seven priority evaluation questions with existing or readily obtainable data. ------- Key Findings Based on the activities and methods described above, IEc found the following: Existing data - combined with new qualitative research (e.g.. interviews') - can provide meaningful indications of the program's successes, challenges, and opportunities. Qualitative research can explore factors that influenced the development of RE projects on CLs; the program's role and contribution to observed outcomes; market drivers and barriers to developing RE projects on CLs; and ways to strengthen the program. Given existing data limitations, it is not currently possible to prove the direct causal impact of the program on the development of RE projects on CLs. or to comprehensively assess the program in its entirety. A lack of comprehensive information about who has accessed the program's resources, and in what capacity, prevents a comprehensive assessment of the program's effectiveness. Evaluation topics and methods that can be supported with existing data include: o An in-depth assessment of the effectiveness of EPA and NREL's joint feasibility studies and the program's liability comfort letters. o Interviews with program partners, industry experts, and other stakeholders about various dimensions of program effectiveness. o A survey to understand how the program interacts with other EPA programs and other federal agencies. o A literature review to match program activities and outputs to high-, medium-, and low-priority barriers to RE development on CLs. o A citation analysis to identify third-party references to the program, its outputs, its partners, or broader industry issues to assess the program's role in addressing barriers. o Application of decision rules to approximate the avoided development on undisturbed lands and possibly the cost impacts of RE projects on CLs. Evaluation topics and methods that cannot be supported with existing data include: o An in-depth assessment of the effectiveness of tools and resources available on the program's website; information provided at presentations, conference sessions, and workshops; assistance from the RE- Powering Response Team; and assistance from EPA Headquarters staff. o Development of robust quantitative measurements for the proportion of stakeholders connecting with the program who later realized RE projects on CLs, or for the total number of RE projects developed on CLs that the program influenced. o Surveys of stakeholders, partners, and/or experts outside of the federal government about various dimensions of program effectiveness and their interactions with the program. Data Improvement Considerations IEc offers the following recommendations and observations to strengthen the program's data collection activities: Major recommendations: o Systematically track users (and the extent of use) of the program's resources, tools, and knowledge products to enable a quantitative evaluation of the effectiveness of these outputs. o Systematically track progress toward developing RE projects on CLs to develop a deeper understanding of how, when, and why CLs progress (or do not progress) through the RE project development process. It appears based on an initial investigation that the program could likely make significant improvements to its current data collection approaches using technology that EPA already has in-house and with limited additional cost. The program may be able to customize the assistance that it provides by collecting additional information about users of its resources, tools, and knowledge products (e.g., state and type of technology). Collecting longitudinal data on specific individuals' use of program resources would require each user to create a separate account, which might be accomplished through online registration or the RE-Powering Response Team. All such approaches should consider Information Collection Request (ICR) requirements, relevant Agency policy, data privacy issues, and security issues in determining how to collect, retain, and use any additional data. Next Steps The Initiative continues to believe that evaluative information will help the Agency guide its actions to better realize environmental and other benefits that RE-Powering investments have to offer. The Initiative plans to pursue in the coming year analysis suggested by the scoping study, although the number of questions and types of information collected and analyzed will depend upon resources and availability. For additional information or comments on this evaluation scoping assessment, please email cleanenergy@epa.gov. ------- |