oEPA
United States
Environmental Protection
Agency
Office of Policy
(1807T)
July 2013
EPA-100-F-13-007
OSWER Risk Management
Program Evaluation Scoping
Project
July 2013
Fact Sheet
Introduction
• The Risk Management Program (RMP) is implemented by the Office of Emergency
Management (OEM) in EPA's Office of Solid Waste and Emergency Response (OSWER).
• EPA and state and local implementing agencies conduct inspections at RMP facilities to
determine compliance with RMP regulatory requirements.
• Because resources for conducting inspections are limited, within the past few years EPA
has begun focusing more attention on high-risk facilities for RMP inspections.
• OEM requested an outcome evaluation of the RMP focusing on the role of inspections.
OEM and ESD agreed that prior to committing to a full evaluation, they would conduct
an evaluation scoping project to: 1) assess whether current data are adequate to
support an evaluation, and 2) identify additional data collection that might be required.
• The project was conducted in two phases. Following the scoping assessment in Phase 1,
EPA determined that the program is not ready for evaluation at the present time. In
Phase 2, EPA developed a Performance Data Improvement Plan to enhance the quality,
accessibility, and usefulness of the data to prepare the program for future evaluation.
• The project team included representatives from: EPA's Evaluation Support Division,
Office of Emergency Management, OSWER Center for Program Analysis, Office of
Enforcement and Compliance Assurance, and contractor Industrial Economics (lEc).
Evaluation Questions
• What effect, if any, do RMP inspections have on facility behavior?
• What effect, if any, do RMP inspections have on the incidence and severity of chemical
accidents at RMP facilities?
• What indications exist, if any, that RMP inspections have a deterrent effect on RMP
facilities that have not been inspected?
• What effect, if any, has the change in RMP inspection strategy (to designate some
facilities as "high risk" and to devote more inspection resources to high-risk facilities)
had on facility behavior and the incidence and severity of accidents?
• Based on the results of the previous question, should EPA consider refining its approach
to defining "high-risk" facilities? Should EPA reconsider the current allocation of
inspection resources between high-risk facilities vs. facilities that are not judged to be
high risk?
Evaluation Scoping Methods
The evaluation team conducted the following activities:
http://www.epa.gov/evaluate
For more information on
completed evaluations at
EPA or the Evaluation
Support Division, visit the
above link.
Refined the program logic model focusing on the role of RMP inspections.
Refined the evaluation questions based on the revised logic model.
Shared the evaluation questions with the ten EPA regions; three provided feedback.
Investigated national EPA databases containing inspection and enforcement data (ICIS)
and accident history data (RMP Info).
Reviewed a sample of inspection reports from four EPA regions.
Reviewed state-level RMP databases provided by two delegated states in EPA Region 4.
Prepared a Scoping Report with findings and overall evaluability assessment.
Prepared a Performance Data Improvement Plan to enhance the quality, accessibility,
and usefulness of the data to support future evaluation efforts.
-------
Key Findings
• EPA could study associations between chemical accidents and RMP inspections with existing data:
RMP Info includes accident histories for facilities that have, and have not, been inspected. Linking the inspection
data in ICIS to the accident data in RMP Info would enable EPA to compare accident histories across inspected and
uninspected facilities. EPA could also examine the accident histories at inspected facilities before and after an RMP
inspection was conducted. This approach would indicate the association, if any, between RMP inspections and
chemical accidents. The ICIS inspection data and RMP accident data can be linked for 80 percent of RMP facilities
at present; the other 20 percent of RMP facilities would be excluded from the analysis pending completion of EPA's
"bridge table" linking the two databases. The ICIS inspection data only go back to FY 2007, limiting the timeframe
of the ana lysis.
• Existing data are not adequate to assess changes in facility beha vior from RMP inspections: Neither
database (RMP Info or ICIS) contains reliable indicators of changes in facility behavior. Limitations include: data
quality concerns regarding the self-reported RMP data, limited information about inspection results, and lack of
information on changes in the behaviors that RMP inspections may influence (e.g., adoption of safer practices). The
best way to assess changes in facility behavior is to conduct follow-up inspections - but this has not been done at
most RMP facilities. Many facilities have never received an RMP inspection, and very few have received more than
one RMP inspection. The lack of follow-up inspection data is a significant evaluability limitation.
• Existing data are not adequate to assess the effects of changes in the RMP inspection strategy:
Assessing the effects of the strategy (to designate some facilities as "high risk" and to devote more inspection
resources to high-risk facilities) on facility behavior is not possible, because RMP Info and ICIS do not provide
reliable indicators of changes in facility behavior. In addition, it would be premature to compare the accident
histories of "high-risk" and other facilities given the short time that has passed since the strategy was adopted, and
the corresponding lack of inspection and accident data for "high-risk" facilities vs. other facilities.
• EPA has not systematically studied the effect of RMP inspections on facilities that are not inspected:
The literature suggests that inspections not only influence inspected facilities, but can also influence uninspected
facilities by increasing their awareness of regulatory requirements and the perceived threat of inspection. While
EPA has studied this issue in general, it has not systematically documented examples that are specific to the Risk
Management Program. Understanding the conditions under which RMP inspections can influence behavior at
uninspected facilities would require further research, including interviews with facility managers and inspectors.
Recommendations
The evaluation team suggests that EPA take steps to improve the quality, accessibility, and usefulness of the data:
• Review the RMP data, notify facilities about potential errors, and verify that facilities submit needed corrections.
• Expand the "bridge table" between RMP Info and ICIS to include more facilities.
• Review accident history and inspection data to establish chronology and account for duplicate records.
• Add new data fields to RMP Info and ICIS to reliably capture changes in facility behavior.
• Ensure that inspections are properly linked to resulting enforcement actions in ICIS.
• Continue to work with EPA's Regional Offices to ensure the quality and consistency of inspection reports.
• Backfill the ICIS database using pre-FY 2007 inspection reports, to the extent feasible.
• Develop a clearer understanding of when and how different regions have implemented the "high-risk" strategy.
• Verify that EPA can link facility risk status (high-priority list) to inspection results and accident history.
• Study and document cases of general deterrence (effects of inspections on uninspected facilities) in the program.
• Consider implementing a pilot study to target a subset of facilities for more frequent inspections.
Contact(s)
• Michelle Mandolia, EPA Evaluation Support Division, mandolia.michelle@epa.QOv
• Shari Grossarth, EPA Evaluation Support Division, Qrossarth.shari@eDa.QOv
• Bill Finan, EPA Office of Emergency Management, finan.bill@epa.QOv
Report Link: http://www.epa.Qov/evaluate/reports/index.htm
Date Completed: July 2013
------- |