United States
                                    Environmental Protection
                                    Office of Policy
July 2013
                   OSWER Risk Management
                   Program  Evaluation  Scoping
                                                                                           July 2013
  Fact Sheet
    The Risk Management Program (RMP) is implemented by the Office of Emergency
     Management (OEM) in EPA's Office of Solid Waste and Emergency Response (OSWER).
    EPA and state and local implementing agencies conduct inspections at RMP facilities to
     determine compliance with RMP regulatory requirements.
    Because resources for conducting inspections are limited, within the past few years EPA
     has begun focusing more attention on high-risk facilities for RMP inspections.
    OEM requested an outcome evaluation of the RMP focusing on the role of inspections.
     OEM and ESD agreed that prior to committing to a full evaluation, they would conduct
     an evaluation scoping project to: 1) assess whether current data are adequate to
     support an evaluation, and 2) identify additional data collection that might be required.
    The project was conducted in two phases. Following the scoping assessment in Phase 1,
     EPA determined that the program is not ready for evaluation at the present time. In
     Phase 2, EPA developed a Performance Data Improvement Plan to enhance the quality,
     accessibility, and usefulness of the data to prepare the program for future evaluation.
    The project team included representatives from: EPA's Evaluation Support Division,
     Office of Emergency Management, OSWER Center for Program Analysis, Office of
     Enforcement and Compliance Assurance, and contractor Industrial Economics (lEc).

Evaluation Questions
    What effect, if any, do RMP inspections have on facility behavior?
    What effect, if any, do RMP inspections have on the incidence and severity of chemical
     accidents at RMP facilities?
    What indications exist, if any, that RMP inspections have a  deterrent effect on RMP
     facilities that have not been inspected?
    What effect, if any, has the change in RMP inspection strategy (to designate some
     facilities as "high risk" and to devote  more inspection resources to high-risk facilities)
     had on facility behavior and the  incidence and severity of accidents?
    Based on the results of the previous question, should EPA consider refining its approach
     to defining "high-risk" facilities? Should EPA reconsider the current allocation of
     inspection resources between high-risk facilities vs. facilities that are not judged to be
     high  risk?

Evaluation Scoping  Methods
The evaluation team conducted the following activities:

 For more information on
 completed evaluations at
 EPA or the Evaluation
 Support Division, visit the
 above link.
      Refined the program logic model focusing on the role of RMP inspections.
      Refined the evaluation questions based on the revised logic model.
      Shared the evaluation questions with the ten EPA regions; three provided feedback.
      Investigated national EPA databases containing inspection and enforcement data (ICIS)
      and accident history data (RMP Info).
      Reviewed a sample of inspection reports from four EPA regions.
      Reviewed state-level RMP databases provided by two delegated states in EPA Region 4.
      Prepared a Scoping Report with findings and overall evaluability assessment.
      Prepared a Performance Data Improvement Plan  to enhance the quality, accessibility,
      and usefulness of the data to support future evaluation efforts.

Key Findings

     EPA could study associations between chemical accidents and RMP inspections with existing data:
      RMP Info includes accident histories for facilities that have, and have not, been inspected. Linking the inspection
      data in ICIS to the accident data in RMP Info would enable EPA to compare accident histories across inspected and
      uninspected facilities. EPA could also examine the accident histories at inspected facilities before and after an RMP
      inspection was conducted. This approach would indicate the association, if any, between RMP inspections and
      chemical accidents. The ICIS inspection data and RMP accident data can be linked for 80 percent of RMP facilities
      at present; the other 20 percent of RMP facilities would be excluded from the analysis pending completion of EPA's
      "bridge table" linking the two databases. The ICIS inspection data only go back to FY 2007, limiting the timeframe
      of the ana lysis.

     Existing data are not adequate to assess changes in facility beha vior from RMP inspections: Neither
      database (RMP Info or ICIS) contains reliable indicators of changes in facility behavior. Limitations include: data
      quality concerns regarding the self-reported RMP data, limited  information about inspection results, and lack of
      information on changes in the behaviors that RMP inspections may influence (e.g., adoption of safer practices). The
      best way to assess changes in facility behavior is to conduct follow-up inspections - but this has not been done at
      most RMP facilities. Many facilities have never received an RMP inspection, and very few have received more than
      one RMP inspection. The lack of follow-up inspection data is a significant evaluability limitation.

     Existing data are not adequate to assess the effects of changes in the RMP inspection strategy:
      Assessing the effects of the strategy (to designate some facilities as "high risk" and to devote more inspection
      resources to  high-risk facilities) on facility behavior is not possible, because RMP Info and ICIS do not provide
      reliable indicators  of changes in facility behavior. In addition, it would be premature to compare the accident
      histories of "high-risk" and other facilities given the short time that has passed since the strategy was adopted, and
      the corresponding lack of inspection and accident data for "high-risk" facilities vs. other facilities.

     EPA has not systematically studied the effect of RMP inspections on facilities that are not inspected:
      The literature suggests that inspections not only influence inspected facilities, but can also influence uninspected
      facilities by increasing their awareness of regulatory requirements and the perceived threat of inspection. While
      EPA has studied this issue in general, it has not systematically documented examples that are specific to the Risk
      Management Program. Understanding the conditions under which RMP inspections can influence behavior at
      uninspected facilities would require further research, including  interviews with facility managers and inspectors.

The evaluation team suggests that EPA take steps to improve the quality, accessibility, and  usefulness of the data:

     Review the RMP data, notify facilities about potential errors, and verify that facilities submit needed corrections.
     Expand the "bridge table" between RMP Info and ICIS to include more facilities.
     Review accident history and inspection data to establish chronology and account for duplicate records.
     Add new data fields to RMP Info and ICIS to reliably capture changes in facility behavior.
     Ensure that inspections are properly linked to resulting enforcement actions in ICIS.
     Continue to work with EPA's Regional Offices to ensure the quality and consistency of inspection reports.
     Backfill the ICIS database using pre-FY 2007 inspection reports, to the extent feasible.
     Develop a  clearer  understanding of when and how different regions have  implemented the "high-risk" strategy.
     Verify that EPA can link facility risk status (high-priority list) to  inspection results and accident history.
     Study and document cases of general deterrence (effects of inspections on uninspected  facilities) in the program.
     Consider implementing a pilot study to target a subset of facilities for more frequent inspections.

     Michelle Mandolia, EPA Evaluation Support Division, mandolia.michelle@epa.QOv
     Shari Grossarth, EPA Evaluation Support Division, Qrossarth.shari@eDa.QOv
     Bill Finan,  EPA Office of Emergency Management, finan.bill@epa.QOv

Report Link: http://www.epa.Qov/evaluate/reports/index.htm

Date Completed: July 2013