EPA 510-R-00-002 Project Contacts Richard Mattick, USEPA/OUST 703/603-7154 Mattick.Richard@epa.gov John Connor, GSI 713/522-6300 ' ' jaconnor@gsi-net.com Scott Murphy, ASTM 610/832-9685 smurphy@astra.org State and Regional Contacts Gilberto Alvarez, USEPA Region 5 Chicago, Illinois David Ariail, USEPA Region 4 Atlanta, Georgia ChetClarke,TNR.CC Austin, Texas Douglas Clay, Illinois EPA Springfield, Illinois Lynn Dail, USEPA Region 6 Dallas, Texas Alan Hancock, USEPA Region 7 Kansas City, Kansas James Humeston, Iowa DNR Des Moines, Iowa Ruth Strauss, NC DENR Raleigh, North Carolina Jo Taylor, USEPA Region 8 Denver, Colorado PaulZahn, Utah DEQ Salt Lake City, Utah fcPROV^ Risk-Based Decision Making Performance Assessment Study Bulletin #2 Study Results and Recommendations for RBDM Program Performance Monitoring Practical measures have been developed for evaluating the impact of Risk- Based Decision Making (RBDM) on state underground storage tank corrective action programs. These measures have been used to evaluate the effect of RBDM on the corrective action programs of five pilot states. In a majority of the pilot states, implementation of an RBDM program resulted in an increase in case closures and a decrease or stabilization in case backlog. Additional findings indicate that these RBDM programs successfully targeted low-risk sites for closure while retaining higher-risk sites for further action. Minor modifications to state program databases will allow for a more detailed evaluation of program performance measures. This document has been funded wholly by USEPA under assistance agreement X825708-01. The infor- mation may not necessarily reflect the view of the Agency, and no official endorsement should be inferred. Introduction The Risk-Based Decision Making (RBDM) Performance Assessment Study has been conducted as a research effort designed to assist state and territorial environmental regulatory agencies with the evaluation of their individual RBDM programs for Under- ground Storage Tanks (USTs). The specific goals of this study were to: i) develop practical, quantitative measures for evalu- ating the impact of RBDM on achieving state agency manage- ment goals, ii) apply these meas- ures to five state RBDM programs to evaluate program performance, and iii) provide general guidelines for other state and territorial envi- ronmental agencies interested in tracking the performance of their RBDM programs. This Bulletin reviews the proce- dures and results of this RBDM Performance Study and presents guidelines for monitoring and evaluating the effectiveness of RBDM programs. In addition, self-reported evaluations of two other state programs are included. The study was funded by the United States Environmental Protection Agency (USEPA) Office of Underground Storage Tanks (OUST) under Assistance Agree- ment #X825708-01 to the Ameri- can Society for Testing and Mate- rials (ASTM). Groundwater Services, Inc. (GSI), of Houston, Texas, has conducted the study. The results have been reviewed by the participating states, USEPA regions, USEPA OUST, and by Partnership in RBCA Implemen- tation (PIRI). Risk-Based Decision Making The USEPA Office of Solid Waste and Emergency Response (OSWER) Directive 9610.17 en- courages all state UST programs to apply RBDM to the corrective action process at petroleum re- lease sites. RBDM is a flexible de- cision management framework that is customized to fit the needs of individual agency programs. When applied to the UST correc- tive action process, RBDM may also be referred to as Risk-Based Corrective Action or "RBCA". The ASTM RBCA Standard E- 1739-95 is one example of an RBDM framework that has been used by states to design or March 2000 RBDM Performance Assessment Bulletin #2 Page 1 of 11 ------- ILLINOIS ENVIRONMENTAL PflOTECTlON AGENCY IOWA DEPARMENT OF NATURAL RESOURCES NCDENR TEXAS NATURAL RESOURCE CONSERVATION COMtSSlON augment their corrective action programs. An RBDM program typically includes three principal activities: Risk-Based Site Prioritization: Pri- oritize sites based on the timing or magnitude of potential im- pacts to human health and the environment. Site-Specific, Risk-Based Remedia- tion Goals: Determine risk-based concentration limits for affected environmental media designed to prevent impacts on human health and the environment. Tier 1 remediation goals repre- sent generic concentration limits, based on conservative default assumptions. Tier 2 and Tier 3 provide site-specific media cleanup limits based on addi- tional site data and more sophis- ticated data analysis. Remedy Selection: Select remedia- tion alternatives, such as re- moval or exposure control, to address site-specific risk drivers. RBDM is a science-based process that offers a clearly defined and consistent basis for site evaluation and remediation. As a result, im- plementation of RBDM corrective action programs is expected to result in increased program effi- ciency and improved risk reduc- tion. RBDM Performance Assessment Study Background Five state environmental regula- tory agencies have participated in this preliminary study to evaluate the effectiveness of their RBDM programs: Illinois: Illinois Environmental Protection Agency, Leaking Un- derground Storage Tank Section Iowa: Iowa Department of Natural Resources, Under- ground Storage Tank Section North Carolina: North Carolina Department of Environment and Natural Resources, Groundwater Section Texas: Texas Natural Resource Conservation Commission, Remediation Division Utah: Utah Department of Envi- ronmental Quality, Division of Environmental Response and Remediation . Each of the selected pilot states has implemented a state RBDM program and expressed interest in evaluating their program per- formance. To provide geographi- cal balance, no more then one state per USEPA Region was se- lected for this study. In addition to the five pilot states, this bulletin summarizes the program per- formance of two states, Michigan and Alabama, which have evalu- ated the impact of their RBDM programs independent of this study. For this study, the pilot states provided input on their individ- ual program goals, the perform- ance measures currently utilized in their state, and the utility and feasibility of the RBDM perform- ance criteria developed for this study. In addition, each pilot state provided program performance data in the form of program data- bases, and assisted with internal program evaluations and inter- pretations of study results. RBDM Program Goal and Performance Measures The overall goal of state UST pro- grams is to protect human health and the environment from re- leases associated with leaking un- derground storage tank (LUST) sites. In order to achieve this goal, March 2000 RBDM Performance Assessment Bulletin #2 Page 2 of 11 ------- the RBDM programs imple- mented by the pilot states ad- dressed in this study shared the following objectives: Risk Reduction: Reduce the human health and ecological risks associated with LUST sites. Expedited Site Evaluation, Remediation, and Closure: Streamline the site assessment process to close sites which do not pose an unacceptable risk to human health or the environ- ment and expedite the remedia- tion of sites with unacceptable risks. Cost Control/Resource Allocation: Reduce the cost of site remedia- tion and closure without com- promising protection of human- health and the environment. Fo- cus available resources on high- risk sites. Reduce the adminis- trative cost of program man- agement. TABLE 1: LIST OF SUGGESTED PERFORMANCE MEASURES FOR EVALUATION OF LUST RBDM PROGRAMS PROGRAM GOAL Risk Reduction Expedited Evaluation, Remediation, Closure Cost Control/ Resource Allocation PERFORMANCE MEASURE Composite Site Classification Profile (see Figure 3) ?/-V Sf £»/ ig3***~ * Compojife CorfeflfuentKS- &T 4.**" Ji -?1"? Cleanup Completed alog No Action Sites Time to Action Plan Approval Remediation Cost TRACKING DESCRIPTION Distribution of risk-based site classification ratings or sum of site classification scores through time Number of corrective ac- tions resulting in case clo- sure urfirieijpf sites currently*" anaeed by Ihe state j ~ *~ *.** ^0. sfeSf A-0* -^ ^ *" " Percentage and overall number of sites not requiring corrective action following risk-based site pjreiall^ ^f evaluation Percente Subs? 0 Time from incident report- ing to state approval of a corrective action plan Deporting w"Hje_c|pstire e Total expense from incident reporting to case closure for the responsible party or for the state reimbursement fund SITE DATA REQUIRED Initial site classification, Current site classification, Site score (i.e., high score for high risk) Ooiuredate L - Closure date, Basis for closure notice,^ t- ' j ipnT?lan approval Sate Incident reporting date, Action Plan approval date I^cidera repoffirtgfdatej" Site remediation cost (including assessment and closure costs), Remediation cost reimbursed by state fund March 2000 RBDM Performance Assessment Bulletin #2 Page 3 of 11 ------- To assist states in evaluating the benefits of their RBDM programs, this study has identified quantita- tive performance measures that correspond to each of the three common RBDM program goals (see Table 1). By tracking these parameters over time for their LUST site population, states can measure progress with regard to risk reduction, expedited site evaluation, remediation, closure, and cost control. The suggested performance measures should be viewed as a menu of options, from which each regulatory authority can select the most appropriate measures for their specific program. These per- formance measures are described in more detail in the prior Per- formance Assessment Study Bul- letin #1 issued in March 1999 (ASTM, 1999). To determine the impact of RBDM implementation on program performance, these measures have been used to evaluate program performance based on data available from each of the pilot states. Evaluation of Pilot State Programs: Findings and Implications The five pilot states each maintain a database of all active and closed LUST sites which have been en- tered into the regulatory process for site remediation. GSI reviewed each database to identify pa- rameters which could be used to evaluate progress toward RBDM program goals. The pilot states implemented RBDM for their cor- rective action programs between 1994 and 1998 (see Table 2). In order to compare program per- formance before and after RBDM implementation, performance has been evaluated for the period of 1990 to 1999. The results of this study support the following general findings with regard to the common pro- gram management goals: Expedited Site Evaluation, Remediation, and Closure: Im- mediately following imple- mentation, of their RBDM pro- gram, 4 of the 5 pilot states ob- served a dramatic spike in case closures per year and a stabili- zation or decrease in case back- log. In general, the average case age at time of closure increased following RBDM implementa- tion. Combined with the in- crease in case closures, this finding indicates that many older cases which have been in the regulatory process for many years are now being addressed. The observed reduction in case backlog is likely to reduce the administrative burden for the regulatory agency, allowing a more efficient allocation of available resources. In the first year of the Utah RBCA Tier 1 program (1995), the number of case closures in- creased by 120%, indicating that a Tier 1 process of generic screening criteria can signifi- cantly impact program per- formance (see Figure 1). Fol- lowing implementation of a RBDM-based corrective action program in Iowa, 77% of RBCA Tier 1 site assessment reports approved by the DNR resulted in case closure and 28% of Tier 2 site assessments re- sulted in case closure, indicating that the RBDM process has been effective at identifying sites for closure or remediation. Key re- sults for individual pilot states are summarized in Table 2. Risk Reduction: Available in- formation regarding risk-based site classification (Texas and North Carolina) indicates that most of the LUST site cases March 2000 RBDM Performance Assessment Bulletin #2 Page 4 of 11 ------- 800 Utah LUST Case Closures Per Year 1990 1991 1992 1993 1994 1995 1996 1997 1998* Year '* = 1998 data is through 10/1/98. FIGURE 1: EFFECT OF RBDM ON LUST CASE CLOSURES IN UTAH 8000, Texas LUST Case Backlog 7000. 6000. 5000, 4000. 3000. 2000. 1000. o High-Risk Sites Low-Risk Sites ,RBCA, Exit Criteria - 1990 1991 1992 1993 1994 . 1995 1996 1997 19: Year North Carolina LUST Case Backlog 1990 1991 1992 1993 1994 1995 1996 1997 199B 1999 FIGURE 2: EFFECT OF RBDM ON LUST CASE BACKLOG IN TEXAS AND NORTH CAROLINA March 2000 RBDM Performance Assessment Bulletin #2 Page 5 of 11 ------- ' !| |,.,!"'||||f ,'' .fl1!1 " ., " , l,"l"'1i I'll! closed by these state RBDM programs are low-risk sites (see Figure 2). These examples dem- onstrate that RBDM programs are effectively meeting the state program objective of closing low-risk cases while retaining higher-risk cases in the regula- tory process for further evalua- tion or remediation. The re- duced backlog of low-risk sites should allow available resources to be more effectively targeted to the higher-risk sites. Cost Control/Resource Alloca- tion: Cost data in the LUST site databases provided by the pilot states were not sufficient to measure the cost impacts of RBDM at this time. However, the significant reductions in case backlog reported by some states clearly corresponds to reduced program costs. An internal cost survey was conducted by the Texas Natural Resource Conservation Com- mission to determine the impact of RBCA implementation on the cost of site remediation. Be- tween 1994 and 1998, remedia- tion/closure costs were reduced by 70% for soil-only sites (me- dian cost reduced to $24,000/site from $80,000/site), and by 58% for low-risk groundwater impact sites (me- dian cost reduced to $107,000 from$250,000/site). Potential Confounding Factors Interpretation of the impact of RBDM on program performance may be complicated by con- founding factors such as the 1998 upgrade deadline, changes in staffing at state regulatory agen- cies, or changes in state funding of site remediation. These factors can also impact the program measures designed to measure the effectiveness of RBDM. To control for these confounding factors, the impact of the RBDM on program performance independent of con- founding factors is best under- stood through the evaluation of multiple performance measures which cover all of the RBDM pro- gram objectives. Results from Other States In addition to the five states evaluated for this study, Michigan and Alabama have independently evaluated the performance of their LUST management pro- grams following the implementa- tion of RBDM. Alabama Depart- ment of Environmental Manage- ment (DEM), UST Corrective Ac- tion Unit implemented an RBDM program in April 1998 and is cur- rently tracking performance. In the first year of the program, Ala- bama DEM saw a reduction of 106 active cases classified as low-risk sites and a corresponding increase of 115 closed cases (4% of the ac- tive'case population). Although recently implemented, the Ala- bama RBDM program appears to be successful in closing low-risk sites (Malaier, 2000). Michigan Department of Envi- ronmental Quality (DEQ), Storage Tank Division implemented an RBDM program in April 1995. In 1996, Michigan DEQ reported a 61% increase in LUST case clo- sures compared to the average case closure rate for 1990 to 1995. In addition, Michigan DEQ achieved a 30% decrease in case backlog from 1995 to 1998. Im- plementation of the RBDM pro- gram resulted in a 24% average reduction in remediation/closure costs for UST sites, representing a $39,000 cost savings per site (Michigan DEQ, 1996). March 2000 RBDM Performance Assessment Bulletin #2 Page 6 of 11 ------- TABLE 2. RESULTS FOR PILOT STATE RBDM PROGRAM PERFORMANCE EVALUATION A »f ~- ' *STATB/AGENCX> '< f % "jf/fl Texas Natural Resource ^ Conservation* *£, Commission, ? Remediation \crier; > 5 Department of Environmental) Duality, Division ' of Environmental Response ^nd Remediation'' / ' 1 "V ** North Carolina 'Department of! Environment Resources, >' * Gg*ttj»djwaier Sec- Jtoa. , Ibfaa "L ; x Department Resources;''"'''*''*/ * Underground "^ -« -Storage Tank ^ *Sechon '- . ,. , Illinois Hlrrfois* p ferff ^X^ 'leaking-^ ^Storage Tank. ^ ^ ^Section - ' RBDM PROGRAM IMPLEMEN- TATION Risk-Based Corrective Action for Leaking Stor- age Tank Sites, January 1994. Exit Criteria, September 1997. Risk-Based Corrective Action Tier 1, September 1995. Risk-Based Corrective Action Tier 2, June 1998. Risk-Based Corrective Action for UST sites, January 1998. Risk-Based Corrective Action, January 1997. Tiered Approach to Corrective Action Objectives, January 1997. DATABASE Responsible Party Remediation Database Leaking : Under- ground Storage Tank Database Incident Management Database UST/LUST Database Leaking Under- ground Storage Tank Database PERFORMANCE MEASURES LUST SITE REMEDIATION/ CLOSURE 46% increase in case closures 1996 to 1997. 31% decrease in case backlog 1994 to 1998. 120% increase in case closures 1994 to 1995. 53% decrease in case backlog 1994 to 1998. 46% increase in case closures 1997 to 1998. 1% decrease in case backlog 1997 to 1999. 134% increase in case closures '94-'96to'97-'99. 14% decrease in case backlog 1996 to 1999. 8% decrease in case closures 1996 to 1997. 8% increase in case backlog 1996 to 1998. RISK REDUCTION Preferential closure of low- risk cases, remediation of higher-risk cases. ID Preferential closure of low- risk Cases, remediation of higher-risk ID ID COST CONTROL 58 to 70% decrease in remediation cost for low-risk soil or groundwater sites. ID ID ID ID NOTE: ID = Insufficient Data General Recommendations for RBDM Performance Monitoring Evaluation of the pilot state data- bases shows that most states track data to provide important infor- mation on the performance of their regulatory program with re- gard to key management objec- tives. However, additional data that would allow for a more com- plete assessment of program per- formance was often available in individual site reports but was not recorded electronically. In many cases, minor modifications of da- tabase parameters may be re- quired to incorporate the quanti- tative performance measures identified in this study (see Table 1). Customizing a state program for the purpose of an RBDM March 2000 RBDM Performance Assessment Bulletin #2 Page 7 of 11 ------- Yean Year 2 Years High \->. Low Risk Category High f- Low Risk Category High f- Low RiskCatagory FIGURE 3: USE OF SITE CLASSIFICATION PROFILE TO TRACK RISK REDUCTION performance assessment involves the following steps: i) using the list provided in Table 1 as a guide, select or create relevant perform- ance measures that address key program objectives; ii) review the current state database to deter- mine whether required data is currently recorded; and iii) mod- ify the current state database to include the missing data fields. The LUST module of "UST Ac- cess," a database system devel- oped by the USEPA Office of Un- derground Storage Tanks (OUST), is an example of a database that contains many of the required data fields and can also be readily modified or queried for evalua- tion purposes. For more informa- tion on UST Access, contact USEPA OUST at the web address provided at the end of this bulle- tin. In addition to these general guidelines, results of this pilot study show that the following database parameters can signifi- cantly impact the ability to track program performance: 1) Basis for Closure: As a sup- plement to the time to closure measure, a "Basis for Closure" field can be used to record the reason that the case qualified for closure. Tracking the reason for case closure would assist in iden- tifying the types of sites being managed efficiently under RBDM, as well as those for which achieving closure is still difficult. In addition, this field will allow a more complete interpertation of the "time to closure" measure. To assist in analysis of this per- formance data, die reason for clo- sure should be based on defined categories, such as those listed on Table 3. 2) Risk Reduction: Surveying the risk-based site classification of the LUST site population over time (as shown on Figure 3) can serve as a convenient measure of risk- reduction. If the RBDM program is effectively reducing risk, the site classification profile should reflect a general shift toward low- risk categories from year to year. However, to measure risk reduc- tion, the site classification system must be based on the magnitude and immediacy of potential im- pacts on site receptors, using crite- ria similar to those employed in the ASTM RBCA site classification system (ASTM, 1995). Classifica- tion systems based on physical site characteristics (e.g., soil type, groundwater velocity, chemical contaminants) do not reflect the change in risk conditions as remediation progresses and risks are mitigated. Furthermore, to quantify risk reduction, both the initial and the current risk classifi- cation must be recorded for each site. The initial risk classification remains fixed; however, the cur- rent risk classification can change as site remediation progresses. March 2000 RBDM Performance Assessment Bulletin #2 Page 8 of 11 ------- TABLE 3. POTENTIAL LUST CASE CLOSURE CATEGORIES CLOSURE CATEGORY No Action Risk Assessment Soil Excavation Monitored Natural Attenuation Active Soil Remedy Active Groundwater Remedy DESCRIPTION , ,."...' K * * ~ r f ! "^ * *="5 Soil and groundwater constituent concentrations are less than generic screening levels; no remedy or monitoring re- quired. Soil and groundwater constituent concentrations are less than site-specific risk-based standards; no remedy or moni- toring required. Site qualified for closure following excavation of affected site soils. Site qualified for closure following monitored natural at- tenuation of affected groundwater. Site qualified for closure following on-site treatment of soils. Site qualified for closure following active groundwater reme- diation. Using this approach, total risk re- duction over time for the full LUST case population can be quantified as illustrated in Fig- ure 3. The site constituent reduction factor (CRF, the ratio of site con- centration to site-specific clean-up standard for specific constituents) can be used as an additional measure of site risk reduction over time. This performance measure requires that both the initial and current site concentra- tions and the applicable site- specific clean-up standards be re- corded for key constituents. However, the CRF may be diffi- cult to interpret at sites where ex- posure control remedies are se- lected. 3) Remediation Cost: Informa- tion on the cost of site remediation is typically recorded in a database separate from the other program performance data. In addition, cost data is often limited to reim- bursed costs for sites eligible for state funded remediation. The addition of a "Total Reme- diation Cost" field to the primary state database would allow an analysis of remediation costs for all sites regulated under the state corrective action program. This cost data can be collected by re- questing an estimated total reme- diation cost, inclusive of site as- sessment and response action costs, as part of the final request for case closure. 4) Specific Constituents: As demonstrated by the recent focus on MTBE at LUST sites, the pres-. ence of fuel oxygenates in groundwater may have a signifi- cant impact on regulatory pro- gram performance of RBDM. Re- cording the specific constituents which exceed remediation goals at each site could serve to illustrate the effect of individual constitu- ents on case closure, remediation cost, or other performance meas- ures. For some states, recording addi- tional program performance data in an electronic database may rep- resent a significant burden in terms of cost and manpower re- quired to obtain, validate, and in- put the data. Clear and simple guidelines for submittal of re- quired site reports can serve to reduce the burden of recording March 2000 RBDM Performance Assessment Bulletin #2 Page 9 of 11 ------- program performance data. For example, standardized reports or summary forms can provide key performance data in standardized locations, facilitating the transfer of this data to the state database. Alternatively, states can request submittal of an electronic data summary which contains key per- formance data formatted for direct transfer to the state database. Conclusions In the majority of pilot states, im- plementation of an RBDM pro- gram resulted in an immediate increase in site closures and a sta- bilization or decrease in case backlog. The reduction in case- backlog represents a decreased administrative burden for the cor- rective action program. Average age at closure generally increased which, combined with the in- crease in case closures, indicates that many older sites are being closed using RBDM. Evaluation of site risk classifications in the backlog population indicates that the RBDM programs are effec- tively targeting low-risk sites for closure while retaining higher-risk for further action. Additional study is needed to determine the impact of RBDM on the remedia- tion and closure of these higher- risk sites. Next Steps As demand for government ac- countability increases, more states will need to utilize performance measures to document program performance and identify oppor- tunities for increased efficiency. In addition, as a result of the Gov- ernment Performance and Results Act of 1993, which requires formal cost/benefit evaluations of many federal government programs, agencies are under increasing pressure to effectively track pro- gram performance. Many state programs are facing similar pres- sures due to state legislative man- dates. As state RBDM programs mature and additional perform- ance data are collected, future bulletins may be issued to track performance assessment efforts and address specific issues that may arise. Future bulletins will be available at: www.epa.gov / OUST / rbdm Additional Information For more information on RBDM programs and their implementa- tion, see the following sources: Websites OUST Risk-Based Decision- Making: www.epa.gov/OUST/rbdm ASTM Standards: www.astm.org RBCA State Policy Issues Database: www.gsi-net.com/RBCAPOL Publications 1. American Society for Testing and Materials, 1995, "Standard Guide for Risk-Based Corrective Action Applied at Petroleum. Release Sites," ASTME-1739-95, Philadel- phia, PA. 2. American Society for Testing and Materials, 1998, "Standard Provi- sional Guide for Risk-Based Cor- rective Action," ASTM PS 104-98, Philadelphia, PA. Materials, 1999, "Risk-Based Deci- sion-Making Performance Assess- ment Study Bulletin #1: Study Background, Potential Performance Measures, and Preliminary Find- ings," Philadelphia, PA. http: / / wwTv.epa.gov/swerustl / rbdm/techimpl.htm 4. Groundwater Services, Inc., 1995, "Tier 2 RBCA Guidance Manual," Houston, Texas, 713/522-6300. 5. Malaier, D. S., 2000, Alabama De- partment of Environmental Man- March 2000 RBDM Performance Assessment Bulletin #2 Page 10 of 11 ------- agement, Personnel Communica- tion. / »« u- T-.-O,-. inn^ in if ^S^Dfa 1996 Impact of 1?95 and 1996 Amendments to Part 213, Leaking Underground Storage Tanks, 1994 PA 451, As Amended," State of Michigan Department of Environmental Quality. 7. USEPA Office of Solid Waste and Emergency Response (OSWER), 1996, "Use of Risk-Based Decision- Making in UST Corrective Action Programs," OSWER Directive March 2000 RBDM Performance Assessment Bulletin #2 Page 11 of 11 ------- ------- |