^tosrx
i o \
\ VlV^ | OFFICE OF INSPECTOR GENERAL
vr
-------
Report Contributors: Manju Gupta
Jeffrey Harris
Jeffrey S. Hart
Danielle Tesch
Abbreviations
APHIS
Animal and Plant Health Inspection Service
AQIM
Agricultural Quarantine Inspection Monitoring
BLS
Bureau of Labor Statistics
CFR
Code of Federal Regulations
DOL
Department of Labor
DOT
Department of Transportation
EPA
U.S. Environmental Protection Agency
ESA
Employment Standards Administration
FMCSA
Federal Motor Carrier Safety Administration
GAO
Government Accountability Office
GPRA
Government Performance and Results Act of 1993
HHS
Department of Health and Human Services
ICR
Information Collection Request
ISS-2
Inspection Selection System
MCMIS
Motor Carrier Management Information System
OECA
Office of Enforcement and Compliance Assurance
OIG
Office of Inspector General
OMB
Office of Management and Budget
OSHA
Occupational Safety and Health Administration
SAMHSA
Substance Abuse and Mental Health Services Administration
SafeStat
Motor Carrier Safety Status Measurement System
USD A
United States Department of Agriculture
WHD
Wage and Hour Division
Cover illustration: Various examples of graphs
(Source: National Center for Education Statistics)
-------
tf£D sr^
s U.S. Environmental Protection Agency 2007-P-00027
June 20, 2007
0* U ¦ O L. I I V11 Ul IIIICI I Lul a I UlCvl
Office of Inspector General
* yi// 9
At a Glance
PRO"*^
Catalyst for Improving the Environment
Why We Did This Review
The objective of this review
was to collect successful
practices from Federal
agencies similar to the
U.S. Environmental Protection
Agency's (EPA's) Office of
Enforcement and Compliance
Assurance (OECA) that
extensively use statistical
methods, including random
sampling, to measure and
ensure compliance and to
monitor regulatory programs.
Background
OECA faces many obstacles
in measuring compliance
across its regulated universe.
These include limited
knowledge of its large
universe, limited resources,
and difficulties in collecting
data from States through
random inspections and other
means. These obstacles have
prevented OECA from
calculating compliance rates
for the populations within its
regulated universe and from
demonstrating changes in
compliance and trends.
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.aov/oia/reports/2007/
20070620-2007-P-00027.pdf
Overcoming Obstacles to Measuring Compliance:
Practices in Selected Federal Agencies
What We Found
Federal regulatory agencies with missions and obstacles similar to EPA use
statistical methods to generate compliance information. They use this information
to monitor their enforcement and compliance programs and demonstrate program
results. These Federal programs extensively use statistical methods to identify and
analyze risk, set goals, develop strategies to manage the most significant risks, and
report their accomplishments. While the programs we reviewed face similar
obstacles as OECA, they use practical approaches to overcome these obstacles that
OECA could potentially apply to its programs.
Other programs apply statistical methods, such as selective random inspections, to
develop and publish compliance and other rates for their regulated populations.
Some programs collect data through national surveys, while others require States
to submit data as a condition of grant agreements. Programs leverage resources by
working with statisticians from other offices within their agencies, as well as with
statisticians from universities and external research centers. Programs found that
having a champion in senior management within their agency is essential to
overcome resistance to change and to adopt new methods.
Programs do not use statistical methods solely for reporting compliance rates.
Programs reported that other benefits include identifying previously unknown
risks, quantifying results, verifying the effectiveness of targeting schemes, and
maximizing limited resources.
What We Recommend
We recommend that the Assistant Administrator for Enforcement and Compliance
Assurance establish a plan of action, with milestones, to incorporate using
statistical methods to demonstrate the results of EPA's enforcement and
compliance strategies. In addition, OECA can coordinate with the in-house
statistical expertise available in EPA's Office of Research and Development and
Office of Environmental Information to help develop statistical models and
evaluate external proposals. The Agency accepted our recommendations.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
^£0Sr%
0\
¦n g WASHINGTON, D.C. 20460
June 20, 2007
MEMORANDUM
OFFICE OF
INSPECTOR GENERAL
SUBJECT: Overcoming Obstacles to Measuring Compliance:
Practices in Selected Federal Agencies
Report No. 2007-P-00027
FROM: Wade T. Najjum ^ ^ /
Assistant Inspector General, OffiCe^of Program Evaluation
TO: Granta Y. Nakayama
Assistant Administrator, Office of Enforcement and Compliance Assurance
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. The report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
EPA managers will make final determinations on matters in this report in accordance with
established resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $440,022.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed upon
actions, including milestone dates. We have no objections to the further release of this report to
the public. This report will be available at http://www.epa.gov/oig.
If you or your staff have any questions, please contact me at (202) 566-0832 or
naiiurn. wade@epa.gov: or Jeffrey Harris, Director for Program Evaluation, Cross-Media Issues,
at (202) 566-0831 or harris.ieffrev@epa.gov.
-------
Overcoming Obstacles to Measuring Compliance:
Practices in Selected Federal Agencies
Table of C
Chapters
1 Introduction 1
Purpose 1
Background 1
Scope and Methodology 3
2 How Federal Agencies Similar to OECA Measure Compliance Programs 4
Other Federal Regulatory Programs with Missions Similar to OECA 4
OECA's Obstacles to Developing Compliance Rates 5
Practices Other Federal Regulatory Programs Use to Overcome
Similar Obstacles 6
Using External Sources to Supplement Limited Knowledge of
Regulated Universe 7
Collaborating to Overcome Resource Limitations 8
Demonstrating Benefits of Statistical Methodologies to
Overcome Resistance 10
Using Grants, ICRs, and Contractors to Collect Data from States 11
Conclusion 12
Recommendations 13
Agency Comments and OIG Evaluation 13
Status of Recommendations and Potential Monetary Benefits 15
Appendices
A Details on Scope and Methodology 16
B Federal Regulatory Agency Programs Reviewed by OIG 18
C Selected Sources 27
D Agency Response 29
E Distribution 31
-------
Chapter 1
Introduction
Purpose
The objective of this review was to collect successful practices from agencies
similar to the U.S. Environmental Protection Agency's (EPA's) Office of
Enforcement and Compliance Assurance (OECA) that extensively use statistical
methods to measure, monitor, and report compliance levels. Specifically, we
sought to answer the following questions:
How have other Federal agencies used statistical methods to measure and
ensure compliance?
How did these other Federal agencies overcome the obstacles OECA currently
faces to using statistical methods to measure compliance?
Background
OECA shares regulatory responsibilities with EPA regions and States. EPA's
expenditures associated with improving compliance were $435 million,
$437 million, and $489 million in fiscal years 2004, 2005, and 2006, respectively.
OECA uses these funds for a variety of program activities, including categorical
grants, civil and criminal enforcement, compliance incentives, and compliance
monitoring.
Two recent Office of Inspector General (OIG) reports1 found that OECA could
not demonstrate changes in compliance levels of the diverse populations within its
regulated universe,2 and that OECA's publicly reported performance measures in
fiscal year 2005 did not effectively characterize changes in compliance or other
outcomes. OECA did not have current and complete data on either the regulated
entities or changes in their compliance status. Also, OECA primarily reported
completed activities (known as outputs or activity counts) rather than the results
of the completed activities (known as outcomes). For example, OECA reported
on the performance measure Number of inspections evaluations conducted by
EPA, but it could not determine the true state of compliance across programs in
the regulated universe.
According to OECA, EPA and States do not have the resources to inspect every
regulated entity for compliance with environmental statutes. States principally
conduct targeted inspections at sites where they expect to find a violation
1 EPA OIG Report No. 2005-P-00024, Limited Knowledge of the Universe of Regulated Entities Impedes EPA's
Ability to Demonstrate Changes in Regulatory Compliance, September 19, 2005; and EPA OIG Report No.
2006-P-00006, EPA Performance Measures Do Not Effectively Track Compliance Outcomes, December 15, 2005.
2 The term universe refers to the total number of facilities or entities subject to Federal statutes or regulations.
1
-------
(facilities with a history of noncompliance). Compliance rates based solely on
targeted inspections are biased and are likely not representative of the overall
compliance in the regulated sector.
In contrast, compliance information derived from inspections based on random
samples can help analyze the regulated community's overall compliance. Over
time, this type of compliance information can also show trends and changes in
noncompliance, which can help OECA identify emerging compliance problems
that harm human health and the environment. Such data are central to making
sound management decisions about strategic planning and resource allocation for
regulatory programs.
The Office of Management and Budget (OMB) has directed OECA to develop
performance metrics that measure compliance results, including statistically valid
compliance rates based on random samples of regulated entities. At a minimum,
OMB would like to see OECA make meaningful progress in expanding the use of
statistical methods. OECA conducted several pilot studies between 2000 and
2004 to generate compliance rates for certain industries (e.g., petroleum refining,
iron and steel manufacturing, and organic chemical manufacturing) and specific
segments of regulated populations (e.g., combined sewer municipalities). OECA
has not widely used compliance rates because of budget shortfalls and other
resource, policy, and methodology obstacles.
In a September 2004 memorandum, OECA cited several obstacles to developing
statistical methodologies that are applicable to larger segments of its regulated
universe. OECA indicated in this memorandum that it:
has limited knowledge about its large regulated universe;
lacks in-house statistical expertise to develop statistical methods and resources
to conduct nationwide inspections;
experiences resistance from States to random inspections; and
experiences challenges coordinating data collection from States.
We discuss OECA's obstacles and practices of other Federal regulatory programs
that address these obstacles in detail in Chapter 2.
In the 2006-2011 EPA Strategic Plan, OECA states that it plans to develop
statistically valid noncompliance rates3 for key populations within its national
enforcement priority areas.4
3 OECA defines statistically valid noncompliance rates as an estimate of the noncompliance for the entire
population of regulated entities from a moderate-sized sample of inspections.
4 OECA's national enforcement and compliance assurance program is responsible for maximizing compliance with
10 distinct Federal environmental statutes dealing with preventing and controlling air pollution, water pollution,
hazardous waste, toxic substances, and pesticides. OECA organizes its work in two components: a limited number
of national priorities that focus on significant environmental risks and noncompliance patterns, and core program
activities that implement the requirements of all environmental laws and programs.
2
-------
Scope and Methodology
OECA requested our assistance in developing statistically valid compliance rate
methodologies for larger segments of its regulated universe. During preliminary
research, we searched for compliance rates that States and other Federal agencies
have calculated and reported. After preliminary analyses, we decided to focus
only on Federal agencies because they are subject to similar challenges and
reporting requirements as OECA. After talking with OECA officials, we decided
to limit the scope of our evaluation to identifying Federal regulatory agencies that
use statistical methodologies for implementing nationwide compliance monitoring
and reporting. We conducted field work from June 2006 through August 2006,
and performed our evaluation in accordance with Government Auditing
Standards, issued by the Comptroller General of the United States.
We reviewed documents and interviewed program managers from selected health
and safety compliance offices. While we initially identified more than five
Federal agencies during preliminary research, we judgmentally limited our review
to five programs that used statistical sampling methods to determine compliance
levels, published compliance rates, and used the rates to manage their programs,
and interviewed program managers from these programs (see Table 2-1 in
Chapter 2).
We also reviewed documents provided by program managers in response to our
questions. We analyzed information from the interviews and documents to
identify practices that would be relevant and useful to OECA. We discuss these
relevant practices in more detail in Chapter 2.
Appendix A provides more details on our scope and methodology, including the
criteria used for selecting Federal agencies and the specific topics we discussed
during the interviews. Appendices B and C provide detailed information about
each program and selected bibliographic sources related to their programs,
respectively.
Prior evaluations addressing OECA compliance measurement include the two
EPA OIG reports cited in footnote 1, which outline OECA's difficulties in
quantifying its regulated universe and demonstrating program outcomes with its
current performance measures.
3
-------
Chapter 2
How Federal Agencies Similar to OECA
Measure Compliance Programs
Federal regulatory agencies that have similar missions and obstacles as OECA use
statistical methods to generate compliance information. They use these methods
to monitor their enforcement and compliance programs and demonstrate program
results. Each Federal program regulates a large nationwide universe composed of
diverse populations. Programs are subject to Federal budgetary constraints and
share enforcement responsibilities with States and regions.
Other Federal Regulatory Programs with Missions Similar to OECA
OECA is responsible for enforcing compliance with regulations that support
EPA's mission of protecting human health and the environment. To ensure
compliance with environmental regulations, OECA conducts enforcement
activities such as criminal and civil enforcement, compliance monitoring,
compliance assistance, and providing compliance incentives. OECA monitors
compliance by collecting information from States and regulated entities, as well
as through inspections.
OECA and these other regulatory agency programs are subject to the Government
Performance and Results Act (GPRA) of 1993, which requires Federal agencies to
develop strategic plans, performance goals, performance measures, and report
performance results. Federal agencies also submit their annual goals and report
the outcomes of their program activities to OMB, emphasizing how program
activities contribute to achieving performance goals.
We reviewed five Federal regulatory programs from four agencies with missions
similar to EPA, e.g., protecting human health and/or safety, and safeguarding
natural resources (see Table 2-1). Most programs we reviewed began
incorporating statistical methods into their regulatory processes to report results in
response to requirements of GPRA. Detailed descriptions of these programs are
in Appendix B.
4
-------
Table 2-1: Program Missions of the Five Federal Regulatory Agencies Reviewed
Agency - Program
Program Mission
United States Department of
Agriculture (USDA), Animal and Plant
Health Inspection Service (APHIS)
Safeguard agriculture and natural resources
from risks associated with the entry,
establishment, or spread of animal and plant
pests and noxious weeds to ensure an
abundant, high-quality, and varied food supply.
Department of Labor (DOL),
Occupational Safety and Health
Administration (OSHA)
Assure the safety and health of America's
workers by setting and enforcing standards;
providing training, outreach, and education;
establishing partnerships; and encouraging
continual improvement in workplace safety and
health.
DOL, Employment Standards
Administration (ESA), Wage and
Hour Division (WHD)
Administer and enforce several Federal labor
laws, including minimum wage, overtime pay,
recordkeeping, and child labor requirements of
the Fair Labor Standards Act.
Department of Transportation (DOT),
Federal Motor Carrier Safety
Administration (FMCSA)
Reduce crashes, injuries, and fatalities involving
large trucks and buses.
Department of Health and Human
Services (HHS), Substance Abuse and
Mental Health Services Administration
(SAMHSA)
Build resilience and facilitate recovery for
people with or at risk for substance abuse and
mental illness.
Source: OIG summary of selected agency Websites and source material
OECA's Obstacles to Developing Compliance Rates
OECA identified several obstacles that inhibit its ability to measure and
demonstrate changes in compliance. Some of the reasons OECA provides for not
developing programmatic compliance rates are:
OECA has limited knowledge of its large regulated universe, so it is difficult
to select an accurate sample and develop a valid compliance rate;
OECA finds national inspection-based rates for large populations cost
prohibitive given its limited resources;
OECA lacks internal statistical resources and expertise to develop sampling
plans, identify sample sizes, and analyze results;
States resist random inspections because they believe random inspections are
not as effective as targeted inspections;
OMB requires Federal agencies to file an Information Collection Request
(ICR) to collect nationwide information from States, which delays and
complicates collecting data from States; and
Differences between EPA and State planning cycles make it difficult to
schedule and complete the inspections needed for national compliance
statistics.
5
-------
Practices Other Federal Regulatory Programs Use to Overcome
Similar Obstacles
The Federal regulatory programs we selected faced similar obstacles as OECA
(see Table 2-2). Agencies, and States with delegated regulatory responsibilities,
have integrated statistical methods into all phases of their program
implementation and do not focus solely on producing and reporting compliance
rates for individual projects. They use statistical models to stratify their large
universes into risk groups and to evaluate the success of their targeting strategies.
They selectively use random sampling to identify risks and to demonstrate
program results.
Table 2-2: Other Federal Regulatory Agencies Have Faced Similar Obstacles
OECA's Obstacles
Did the Federal Regulatory Agency Face a Similar
Obstacle?
APHIS
OSHA
WHD
FMCSA
SAMHSA
Limited Knowledge of
Large Regulated
Universe
Yes
Yes
Yes
Yes
Yes
Limited Resources
Inspection Resources
Statistical Resources
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
States' Resistance to
Random Inspections
N/Aa
Yes
N/Aa
Yes
Yes
Data Collection from
States
OMB Information
Collection Request
(ICR) Restrictions
Differences between
EPA and State
Planning Cycles
Yes
N/A
Yes
No b
Yes
N/A
Yes
No b
Yes
No b
a Federal agents conduct all of the inspections for APHIS and WHD. However, these programs
also had to overcome resistance to including random inspections from management and agents.
b OHSA, FMCSA, and SAMHSA did not mention planning cycles as challenges to collecting data
from States.
Source: OIG analysis of interviews with APHIS, OSHA, WHD, FMCSA, and SAMHSA
While we have not evaluated the validity of the statistical methods used by the
other regulatory programs, the Government Accountability Office (GAO), OMB,
and other external evaluators have reviewed these compliance programs and
methods; they have taken no exceptions to using these statistical methods.
We discuss the practices the programs use to overcome obstacles cited by OECA
(see Table 2-3 and following sections). We provide additional details about each
6
-------
of the programs we reviewed in Appendix B. We also list selected sources for
each of these programs in Appendix C.
Table 2-3: Summary of Practices Used by Other Regulatory Agencies to
Overcome OECA's Cited Obstacles
OECA's Obstacles
How Other Agencies Overcame Similar
Obstacles
Limited Knowledge of Large
Regulated Universe
Supplemented knowledge of the regulated
universe from external sources
Limited Resources: Inspection
Resources and Statistical
Resources
Collaborated with States in sharing inspections
responsibilities
Collaborated with statistical resources in other
parts of the agency, as well as external sources
for statistical expertise
States' Resistance to Random
Inspections
Demonstrated additional benefits of statistical
methodologies to overcome resistance
Obtained support and a champion at agency
leadership level
Data Collection from States: OMB
ICR Restrictions and Differences
in EPA and State Planning
Cycles
Collected data from States through grants,
ICRs, and contractors
None of the agencies mentioned differences in
planning cycles as a barrier to collecting data
from States
Source: OIG analysis of interviews with APHIS, OSHA, WHD, FMCSA, and SAMHSA
Using External Sources to Supplement Limited Knowledge of
Regulated Universe
Similar to OECA, the agencies we reviewed each regulates a large universe.
Because these programs also regulate large numbers of industries and facilities
that constantly change in size, the exact size of their universe is often unknown.
They supplement knowledge of their universe by obtaining information from
other parts of their agencies, as well as from States that share enforcement and
compliance responsibilities, and other available reliable databases. Some
agencies require this information from States as part of grant agreements. Some
agencies use outside sources (e.g., Dun & Bradstreet) and external contractors
(e.g., universities and research groups) to survey their regulated universe and
develop a compliance baseline for each universe.
OSHA uses DOL's Bureau of Labor Statistics (BLS) database, as well as
Dun & Bradstreet, for information on its regulated universe. OSHA's universe
consists of approximately 7 million private workplaces in the United States.
OSHA sends out an annual survey to 80,000-100,000 work sites along with
BLS' annual survey to collect site-specific information. The survey helps
OSHA identify establishments that have a higher likelihood of noncompliance.
7
-------
Every year OSHA submits an ICR to OMB to collect site-specific injury and
illness data from regulated entities. OSHA uses its administrative
recordkeeping rule as justification for OMB's approval of its ICR.
FMCSA has information on most of its regulated universe of large trucking
and carrier companies because Federal regulations require every interstate
trucking company to obtain a DOT registration number. Approximately half
of States' motor vehicles departments require a DOT registration number
before issuing license plates to large trucks or carriers.
SAMHSA identifies its universe of retail establishments selling tobacco
products by requiring States to prepare lists of such retailers. SAMHSA
requires this information as part of its grants to States for program
implementation.
APHIS collaborates with the Bureau of Customs and Border Protection to
identify its universe of passengers and cargo entering the United States.
APHIS also collaborates with field office staff to help identify all possible
pathways and their respective sources of risk. APHIS defines pathway as a
route through which agricultural risks enter into the United States, e.g., air
carriers, trucks, cars, and ships.
WHD uses several sources to identify the variety of populations within its
universe that are subject to the Federal Fair Labor Standards Act regulations.
WHD uses data from BLS, State licensing offices, and commercial databases,
including Dun & Bradstreet, for conducting statistically valid, investigation-
based compliance surveys to determine industry compliance rates.
Collaborating to Overcome Resource Limitations
Similar to OECA, the other Federal programs we reviewed have resource
limitations, in terms of their Federal budgets, staff, and in-house statistical
expertise. However, these agencies leverage their resources through collaboration
with their State partners and other stakeholders to develop statistically valid
compliance information. Four of the programs use statistical expertise found in
other parts of their agencies, e.g., other programs within their agency, field
offices, and research centers. Three programs also use external statisticians from
universities and independent research centers.
FMCSA provides current monthly compliance and safety data on its Website.
It requires States to input inspection data into FMCSA's Motor Carrier
Management Information System (MCMIS) database as a condition of the
grants it provides to States. All States provide the data in accordance with
uniform data standards for crash and accident data. FMCSA developed these
standards in collaboration with the National Governors' Association and the
Commercial Vehicle Safety Alliance. States conduct the majority of roadside
8
-------
inspections and submit inspection data. FMCSA has also trained some State
agents to conduct compliance reviews. FMCSA uses the statistical expertise
of DOT's John A. Volpe Research Center for statistical models and data
analyses.
APHIS collaborated with the former U.S. Customs Service to develop and test
the initial Agriculture Quarantine and Inspection Monitoring (AQIM)
methodology. APHIS uses the statistical resources of the USDA National
Agriculture Statistical Services to review agriculture inspection data. APHIS
also works with its field personnel to enumerate risks to agricultural and
natural resources from each pathway. Federal inspectors conduct all regularly
scheduled and additional random inspections associated with AQIM activities.
OSHA uses the resources of statisticians from BLS, in addition to the
occupational injury and illness data submitted to BLS by regulated
establishments. Further, with a few exceptions, OSHA uses its limited
inspection resources to inspect only in those States without State Plan
certification. OSHA relinquishes its inspection authority to the States that
receive certification. The Directorate of Cooperative and State Programs (a
separate group within OSHA) oversees the State Plan States. It has delegated
the authority for monitoring State Plan States to OSHA's 10 regional offices.
Although States are not subject to GPRA, OSHA works collaboratively with
States to develop State-specific multi-year and annual strategic plans. OSHA
requires submission of these plans by States as a part of their annual
applications for OSHA's State Plan grants.
WHD uses its resources to enforce regulations under the Federal Fair Labor
Standards Act, whereas States enforce their individual State labor laws.
Federal inspectors conduct all Federal inspections. WHD uses the University
of Tennessee's Construction Industry Research and Policy Center to analyze
compliance data and provide other statistical services. WHD conducts
national surveys to develop a baseline of compliance information for selected
industries. WHD publishes these compliance rates by region and industry.
SAMHSA provides inspection resources to States to conduct random
inspections and develop lists of retailers selling tobacco. SAMHSA develops
and provides training and statistical models to States. SAMHSA requires
States to use approved protocols for sample selection. States submit
compliance data as a condition of the annual grant. SAMHSA collaborates
with other stakeholders such as retailers, community groups, and parents by
providing training materials, encouraging alertness during State inspections,
and encouraging accuracy in State reporting.
Collaboration with States, internal agency departments, other Federal agencies,
and external stakeholders helped the agencies we reviewed overcome multiple
resource limitations. The programs we reviewed emphasize the need for
9
-------
collaboration and cooperation with States in using uniform data standards and
inspections protocols. Collaboration with States can also leverage resources for
inspections. These programs also emphasize the need for collaboration with other
agency offices in developing statistical methods and analyzing data.
Within EPA, we identified statistical resources from the Office of Research and
Development and the Office of Environmental Information that can assist OECA
in developing statistical models and in evaluating methods proposed by external
statisticians.
Demonstrating Benefits of Statistical Methodologies to Overcome
Resistance
According to OECA, States resist changing their methodologies from targeting
facilities with known or suspected violations to including random selections of
facilities for inspections. States anticipate that some randomly selected sites may
not have violations, and that they use their limited resources more efficiently if
they go after facilities with known or suspected problems.
All the programs we reviewed encountered initial resistance to random
inspections from senior management, field staff, and/or States. Most programs
mentioned two significant factors that helped in overcoming the resistance:
(1) demonstrating the additional benefits of statistical methods, and (2) having a
champion in the agency leadership ranks.
The programs overcame resistance to random sampling by demonstrating
additional benefits of conducting random inspections. These benefits include
identifying previously unknown risks, verifying sources of risk, showing
differences in compliance among regions and industries, and quantifying program
results.
OSHA, WHD, FMCSA, SAMHSA, and APHIS quantified risks to human
health, safety, and agricultural resources by including statistical methods, such
as additional random inspections, in the inspection process. WHD, FMCSA,
and APHIS mentioned that they overcame resistance, and won management
support for random inspections, by demonstrating that random inspections can
verify known risks, identify unknown risks, demonstrate the status of
compliance for the regulated universe, and identify regional or industry
differences in compliance levels. Recognizing that differences exist between
regions and industries allows programs to develop management solutions to
improve compliance specific to regions or industries. They were better able to
identify the regional nature of certain problems, to focus limited resources on
areas that need the most compliance monitoring and assistance. For example,
WHD initially focused limited resources on three industries with the largest
number of low-wage workers and a high incidence of noncompliance with
minimum wage and overtime regulations. For these industries, WHD
10
-------
identified the causes of noncompliance and developed strategies to address
differences in regional compliance rates.
APHIS and WHD demonstrated to inspectors (who target entities based on
past compliance history and professional judgment), that statistical methods
can confirm their "gut feelings" or suspicions. APHIS and WHD also showed
inspectors that statistical methods help quantify the improvements in local
compliance levels resulting from their enforcement activities.
Championship and commitment from agency leaders also helped the agency
programs overcome the initial resistance to incorporating random inspections into
the inspection process. A champion in senior management can advocate using
statistical methods to States and other program offices, reallocate existing
resources for developing and implementing statistical methods, create
opportunities for collaboration with internal and external stakeholders, and
facilitate data collection efforts.
All five of these programs conduct randomly sampled inspections in conjunction
with complaint-based inspections and inspections of known noncompliers. The
programs mentioned that combining targeted inspections with randomly sampled
inspections helped them use limited enforcement resources more effectively. For
example, OSHA allocates 55 percent of its resources to conducting inspections
from statistical samples, and the other 45 percent to conducting inspections based
on complaints and prior knowledge of violations. All the programs generate the
rates that are most significant to meet their program mission and goals. They do
not develop compliance rates for all of the populations within their universe.
They develop a limited number of rates for populations that they select based on
risk assessment.
Programs we reviewed stress the need to set realistic goals when applying
statistical methods to populations within their large universes governed by
complex regulations. The programs initially conducted random samples or
national surveys on a few areas that had high risk, or widespread incidence of
noncompliance. After building national baselines, these agencies selected
additional industries or sectors for which to conduct random inspections and
surveys, and to develop statistically valid compliance rates. WHD found that
annual surveys do not allow sufficient time to analyze the results of surveys,
determine the causes for noncompliance, develop and implement interventions to
improve compliance. As a result, WHD changed its survey strategy to conduct
followup reviews in 5-year intervals.
Using Grants, ICRs, and Contractors to Collect Data from States
OECA needs the compliance data collected by States to create nationwide
compliance statistics. According to OECA, States conduct most of the
inspections at regulated facilities for EPA's delegated programs, but the applicable
11
-------
statutes do not require States to submit data from all inspections to OECA. The
data OCEA receives from States are also of an inconsistent quality. Specifically,
OECA listed two obstacles related to collecting inspection and compliance data
from States:
The need to file an ICR with OMB delays and complicates coordination with
States; and
Differences in EPA and State planning cycles make it difficult to schedule and
complete the inspections needed for a national compliance rate within a year.
The agencies we reviewed had a number of ways of gathering the data needed to
produce compliance rates.
OSHA uses an administrative record-keeping rule that justifies the need for
OSHA to gather injury and accident data from establishments. OSHA submits
an ICR to OMB annually to collect such site-specific data from regulated
entities for managing its compliance program. OSHA also requires annual
goals and performance reports from States as a part of the States' grant
applications. FMCSA and SAMHSA require States to submit data and/or
compliance rates as part of grant agreements. WHD does not need an OMB-
approved ICR to conduct investigation-based surveys because the statute
authorizes WHD to conduct such investigations. WHD uses a university
research center to conduct the surveys to collect and analyze data.
None of the agencies mentioned differences between Federal and State
planning cycles as a challenge to scheduling or completing inspections needed
for compliance statistics. These agencies emphasized that cooperation and
collaboration with States and grant agreements are effective ways to obtain
State compliance data.
Using record-keeping rules, data-reporting requirements in grant agreements,
ICRs, and outside contractors allows agencies to obtain the data needed for
calculating rates (including compliance rates) that identify risks and demonstrate
program outcomes. Agencies that collect data through grant agreements can also
require that the data meet certain standards, which can help to build a reliable
database. Most programs provide funding to States through grants to implement
compliance and enforcement programs. They utilize the grants agreements as
instruments to obtain compliance data from States.
Conclusion
Based on our review of five comparable Federal regulatory programs, we believe
that OECA can overcome its stated obstacles to develop statistical estimates of
compliance. Other agencies have encountered similar obstacles, but have adopted
statistical methods to manage and monitor their compliance programs. Use of
random samples can assist OECA in risk assessment by identifying previously
12
-------
unknown sources of risk, as well as help in evaluating the effectiveness of its
inspections targeting. OECA can use the results of statistical methods to monitor
and demonstrate the results of compliance and enforcement strategies.
OECA needs to commit to a practical approach and develop a plan of action with
milestones. OECA can begin by establishing a set of incremental goals for a few
areas of high environmental risk. Steps can include updating the knowledge of
the populations in its universe, developing statistical samples for the selected
areas, collaborating with and overcoming resistance from States, developing tools
to collect data from States (e.g., grant requirements, ICRs), and requiring reliable
compliance data from States. OECA can obtain in-house statistical support from
EPA's statisticians in the Office of Research and Development and the Office of
Environmental Information in developing statistical models and evaluating
methods proposed by external statisticians.
Recommendations
We recommend that the Assistant Administrator for Enforcement and Compliance
Assurance:
2-1 Establish a plan of action with milestones to incorporate using statistical
methods to demonstrate the results of EPA's enforcement and compliance
strategies.
2-2 Coordinate with the in-house statistical expertise available in EPA's Office
of Research and Development and Office of Environmental Information to
help develop statistical models and evaluate external proposals.
Agency Comments and OIG Evaluation
OECA accepted both of our recommendations; we have included their comments
in Appendix D. Based on OECA's comments, we made changes to the report as
appropriate.
In response to the first recommendation, OECA stated that it has committed to
OMB to expanding the use of statistical methods for specific national priority and
problem areas. OECA is currently conducting two related reviews in order to
develop an action plan for OMB. The reviews relate to strategies for national
priorities and problem areas for the fiscal years 2008-2010 planning cycle and
revision of compliance objectives to focus on national priority and problem areas
rather than on program tools. In response to the second recommendation, OECA
agreed to examine the level of expertise and the types of services that EPA's in-
house statisticians from the Office of Research and Development and the Office
of Environmental Information can provide.
13
-------
OECA's commitment to expand the use of statistical methods to focus on national
priority areas is an encouraging first step. We have provided examples of
approaches other agencies have taken to overcome similar difficulties as OECA
has faced. OECA can obtain more details from these sources and potentially
apply these approaches to its programs. OECA should incrementally develop a
strategy for expanding the use of statistical methods for management of the core
program activities where feasible. In OECA's written response to this report, we
will be looking for (1) a Corrective Action Plan with milestonespotentially a
reiteration of actions previously committed to OMB, and (2) information
regarding the status of compliance in core program activities.
14
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Claimed
Amount
Agreed To
Amount
2-1 13 Establish a plan of action with milestones to
incorporate using statistical methods to
demonstrate the results of EPA's enforcement and
compliance strategies.
2-2 13 Coordinate with the in-house statistical expertise
available in EPA's Office of Research and
Development and Office of Environmental
Information to help develop statistical models and
evaluate external proposals.
Assistant Administrator for
Enforcement and
Compliance Assurance
Assistant Administrator for
Enforcement and
Compliance Assurance
1 0 = recommendation is open with agreed-to corrective actions pending;
C = recommendation is closed with all agreed-to actions completed;
U = recommendation is undecided with resolution efforts in progress
15
-------
Appendix A
Details on Scope and Methodology
Criteria Used to Select Federal Regulatory Agencies
During preliminary research, we searched for compliance rates reported by State and Federal
agencies. We identified 18 States that reported compliance rates. We surveyed seven of these
States to determine if State agencies use statistical methods to measure compliance. We found
that few States use statistical methods for measuring compliance and therefore did not include
States in our evaluation. Additionally, we found that OECA was already collecting such
information from States. After talking with OECA officials, we limited the scope of the
evaluation to identifying successful practices from other Federal agencies.
While we initially identified more than five Federal agencies during preliminary research, we
used the following criteria to limit our scope to those Federal regulatory agencies or programs
that are most applicable to OECA's enforcement situation:
The agency/program has regulatory responsibilities for enforcing/assuring compliance with
statutes/regulations (can be within a single program area or multiple program areas);
The agency/program has a mission related to human health, safety, or the environment;
The agency/program has a large regulated universe;
The agency/program has a national scope focused across States and/or regions;
The agency/program uses (or has used) statistical sampling methods for determining
compliance with statutes/regulations;
The agency/program has developed/published compliance rates; and
The agency/program uses compliance rates as an outcome measure to manage the
agency/program.
We judgmentally selected five Federal regulatory agencies for further review (see Table 2-1).
Interviews with Program Managers
We interviewed program managers from five Federal agency programs to gain a better
understanding of the statistical methods they use, the purpose for which they use the methods,
and how they use statistics for measuring and reporting on compliance, including developing
compliance rates. We requested information on the following topics from all of the
agencies/programs:
Background Information: Background on what initiated the need for developing
statistically valid rates, knowledge of the composition and size of the regulated universe, and
shared enforcement/ inspection responsibilities with States.
Resources Used: Resources used in the development of the statistical methodology;
statistical expertise used (in-house or external contractors); full-time equivalents, money,
time, etc.; and resources needed to calculate the rate on a continuous basis.
16
-------
Description: Description of the methodology, data sources, and data quality.
Benefits and Uses: How does the agency use the statistically valid rates, what benefits has
the agency experienced using the methodology, and what lessons have been learned from
using the methodology.
Opportunity Costs: How does the agency balance the need for targeted sampling with the
need for statistically valid sampling.
External reviews (or related reports): Have the agency's methodologies been reviewed.
We also reviewed documents that the program managers provided in response to our questions.
We analyzed information from the interviews and documents to identify practices that would be
relevant and useful to OECA. We discuss these relevant practices in Chapter 2.
Analysis of Interview Responses
We prepared a table that compared interview responses for 17 variables among the five Federal
regulatory agencies we interviewed. We shared our analysis of interview responses with Federal
agencies and requested that they verify the information we are including in this report. This
information is available in Appendix B.
Scope Limitations
The purpose of this project was to focus on the statistical methods other regulatory agencies use
for monitoring and reporting compliance. We did not review the management controls of OECA
or the other agencies. We relied on information from interviews with program managers,
documents provided by the managers, and other resources available on the identified Federal
agencies' Websites. While we confirmed the accuracy of our information with each individual
agency, we did not independently verify this information or evaluate the validity of their selected
statistical methods. However, GAO, OMB, and other external evaluators have reviewed these
compliance programs and methods; they have taken no exceptions to the use of these statistical
methods.
As coordinated with OECA, we did not evaluate OECA's current methodology for developing
compliance rates; therefore, we also did not compare its methodology with methodologies from
other agencies. We selected regulatory agencies with missions related to human health, safety,
and the environment, but did not determine which statistical methods for measuring compliance
might be most suitable for OECA.
17
-------
Appendix B
Federal Regulatory Agency Programs
Reviewed by OIG
We provide below brief descriptions of the Federal regulatory agency programs we reviewed.
Table B-l provides more details on each program.
United States Department of Agriculture,
Animal and Plant Health Inspection Service, Plant Protection and Quarantine
APHIS calculates two compliance rates: the "actual" and the "predicted" entry rates of pests and
diseases through a pathway. APHIS bases the actual entry rate on programmed inspections and
the predicted entry rate on additional random inspections conducted through APHIS' AQIM
activities. APHIS initiated its effort to use statistical methodologies after Congress passed
GPRA in 1993. The purpose of conducting AQIM activities is to identify the pathways that pose
the most risk of introducing pests or disease to U.S. agricultural resources. APHIS works with
field personnel in performing the risk assessments
Department of Labor,
Occupational Safety and Health Administration
OSHA calculates and publishes national injury and illness rates. OSHA requires regulated
establishments with more than 40 employees to submit injury and illness rates annually; OSHA
conducts random inspections or compliance reviews of regulated establishments with rates
higher than two times the national average. OSHA also provides compliance assistance to
establishments out of compliance with regulations. OSHA monitors the progress of 10,000-
11,000 core establishments to measure injury and illness rates over time. OSHA selects
establishments with injury and illness rates higher than two times the national average for
random inspections.
Department of Labor,
Employment Standards Administration, Wage and Hour Division
WHD measures compliance of establishments with Federal labor laws in order to ensure workers
receive the wages due to them. For over a decade, WHD has used investigation-based surveys to
accurately project compliance rates in a number of low-wage industries affecting a range of
employees. Statistically valid investigation-based surveys (i.e., investigations of establishments
that are not identified by employee complaint, anecdotal evidence, or investigator experience)
provide an unbiased assessment of compliance.
The knowledge gained through statistically valid surveys helps WHD leadership make informed
decisions, in particular, in determining priority areas for directed enforcement and in gaining
insight into effective intervention tools. Survey findings resulted in national attention to specific
industries and particular vulnerable employees within those industries (e.g., youth workers, low-
18
-------
wage workers in particular industries). WHD also uses statistically valid surveys for reporting
purposes to demonstrate the program's impact on compliance.
Department of Transportation,
Federal Motor Carrier Safety Administration
FMCSA uses a number of performance measures for compliance reviews, roadside inspections,
and traffic enforcement. In addition to measuring a large truck-related fatality rate, FMCSA
determines compliance with individual Federal Motor Carrier Safety Regulations and individual
Hazardous Waste Regulations with randomly selected roadside inspections. Data and
information from these inspections assist FMCSA in selecting companies for on-site compliance
reviews.
FMCSA selects for inspections companies that are most likely to have crashes, based on
statistical analyses of performance statistics. It also selects companies for other reasons such as
complaints, carrier requests, etc. FMCSA quantifies risks to human safety by including
statistical methods and using the SafeStat Inspection Selection System (ISS-2). The ISS-2 is an
automated, data-driven system that measures the relative safety fitness of interstate motor
carriers using on-the-road safety, enforcement, and compliance review data. It provides relative
ranking of motor carriers based on performance. The SafeStat system analyzes the prior
30-month history of State reported crashes, roadside inspections, compliance reviews,
enforcement cases, and census data in the Motor Carrier Management Information System
database. The data is time and severity weighted. FMCSA uses the results of these analyses as a
tool in prioritizing motor carriers for compliance reviews and increased roadside inspections.
Department of Health and Human Services,
Substance Abuse and Mental Health Administration
SAMHSA requires States to conduct unannounced random inspections of tobacco retailers to
determine compliance with regulations prohibiting the sale of tobacco products to individuals
under the age of 18. States report to SAMHSA the noncompliance rates, i.e., retailers that sell
tobacco to minors. The Synar Amendment requires that compliance rates be statistically valid.
SAMHSA then calculates a national noncompliance rate by assigning weights, based on State
population, to the State noncompliance rates. Random inspections and reporting of the results is
a grant requirement under the Substance Abuse Prevention and Treatment block grants States
receive.
SAMHSA has an administrative record-keeping rule that outlines its need for data to manage its
programs. SAMHSA analyzes the data in order to measure results and adjust compliance
strategies.
19
-------
Table B-1: Summary of Federal Regulatory Agency Programs Reviewed by OIG Evaluation Team
ro
o
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
Agency
Performance
Goal(s)
Mitigate and reduce
agricultural risk
entering into the United
States
Prevent job-related
illnesses, injuries, and
fatalities
Ensure workers receive
the wages due them
according to Federal laws
Reduce large truck-related
fatality rate by 41 % from
1996 to 2008, resulting in
a 2008 rate of 1.65
fatalities per 100 million
truck vehicle miles
traveled
Restrict sale of tobacco
products to minors
Goal: Attain
compliance by 80% of
tobacco retailers, by
reducing non-
compliance below 20%
Agency
Performance
Measure(s)
Actual entry rate of
pests and diseases
through a pathway
Predicted entry rate of
pests and diseases
through a pathway
Injury rate
Illness rate
Compliance rates of
establishments
withholding fair wages due
to employees
Large truck-related fatality
rate
Compliance of individual
commercial motor
vehicles and drivers with
Federal Motor Carrier
Safety Regulations and
Hazardous Materials
Regulations
Noncompliance rate of
retailers who sell
tobacco to minors
Who Conducts
Inspections
In 2003, inspection
responsibilities were
transferred to the
Department of
Homeland Security's
Bureau of Customs
and Border Protection
State inspectors in
States with approved
State Plans; Federal
inspectors in all other
States
Federal investigators
conduct the investigation-
based statistical surveys
to determine compliance
with Federal labor laws
FMCSA provides funds to
States through grants and
requires States to enter
safety and crash data
from inspections into the
MCMIS database
FMCSA conducts most of
the compliance reviews of
trucking companies, but
some State officials are
trained to conduct
compliance reviews
States conduct
inspections of retailers
selling tobacco that are
also accessible to
minors
SAMHSA provides
funds for inspections
through Substance
Abuse Prevention and
Treatment block grants
Role of the
States
No State involvement
in the AQIM inspection
or inspection
monitoring processes
Conduct inspections in
States with approved
State Plans
Feed inspection data
into Federal data
collection system
Develop strategic plans
and present annual
plans, as part of grant
applications to OSHA
States enforce their
individual State labor
laws; WHD may refer
complaints to state labor
agencies if appropriate.
States conduct most of
the approximately
3 million yearly roadside
inspections
FMCSA provides funds to
States through grants and
requires States to enter
safety and crash data
from inspections into the
MCMIS database
States implement the
program, report data
and results to SAMHSA
States develop rates for
known universe of
tobacco retailers and
submit them to
SAMHSA annually
States compile the
universe of retailers for
their respective States
-------
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
Oversight of
States
Not applicable - only
Federal agents enforce
these laws
Three meetings per
year between Federal
OSHA and State Plan
States
Not applicable - only
Federal agents enforce
Federal laws
The National Governors
Association and
Commercial Vehicle
Safety Alliance developed
uniform data standards
States use standardized
computer software
Provides training to State
inspectors
SAMHSA monitors
State inspections
Reduction in grant
funds is the penalty for
not meeting goals
Center for Substance
Abuse and Prevention
has an internal policy to
monitor the States
once every 3 years for
onsite inspections
Regulated
Universe
Pathways through
which risk can enter
into the agriculture in
the United States
Consists of 10 major
pathways, i.e., ways
passengers and cargo
enter into the United
States: air carriers,
trucks, cars, ships, etc.
Approximately 7 million
private workplaces in
the United States
Universe includes a
variety of industries with
low-wage hourly workers
such as agriculture,
healthcare, garment
manufacturing, grocery
stores, nursing homes,
and restaurants
Approximately 685,000 to
700,000 interstate trucking
companies
States and territories
compile lists of "eligible
outlets," i.e., retailers
that sell tobacco and
are accessible to
minors
Full universe may be
unknown. Rates
represent the universe
of known eligible outlets
In response to the GAO
review, Center for
Substance Abuse and
Prevention now
requires States to con-
duct a study to validate
their retailer lists for
coverage and accuracy
Types of
Inspections
AQIM is an
inspectional process
where random samples
are customized for
each of the pathways
AQIM is conducted in
addition to regular
inspection activities
Conducts safety and
health inspections
Targeted inspections
limited to establish-
ments with more than
40 employees
By law OSHA must
have administratively
neutral selection criteria
45% of inspections are
unprogrammed (from
complaints, referrals,
fatalities, etc.)
55% of inspections are
programmed
Conducts national
investigation-based
statistical surveys to
monitor national, regional
and local progress with
labor regulations
Conducts directed and
complaint-driven
investigations
Approximately 75% of
investigations are
complaint-driven, 21 %
focus on low-wage
industries, and 8-19%
focus on recidivism
FMCSA conducts
roadside inspections and
compliance reviews
FMCSA stratifies
interstate trucking
companies based on the
data on safety records into
four groups
The companies most likely
to have crashes are
selected for inspections
and compliance reviews
SAMHSA requires
States to conduct
inspections from
random samples
throughout the year
Inspections use minors
that attempt to buy
tobacco without any
identification or without
valid identification
States may conduct
additional targeted
inspections for
improving compliance
-------
ro
ro
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
Targeted
Population
Customized random
samples for each of the
pathways posing
agricultural risk
AQIM Handbook
outlines how to
develop targeting plans
for each of the
pathways
Establishments in
selected industries with
more than 40
employees
Industries targeted
based on injury and
illness rates reported
by BLS
OMB limits OSHA's
data collection to
establishments with
more than 40
employees
Excludes construction
industry
WHD develops targeting
strategies for industries/
companies with high
levels of noncompliance
based on information from
surveys
Industries that employ a
large number of low wage
hourly workers and low
compliance rates (as
suggested by WHD
enforcement statistics)
Businesses that employ
child labor
Interstate trucking
companies most likely to
be involved in future
crashes based on their
safety statistics in the
SafeStat database.
Companies with no data in
the MCMIS database
Retail establishments
are selected from
random samples
States may target
retailers suspected of
noncompliance in
addition to the random
inspections required by
SAMHSA
How
Establishments
are Selected for
Inspection
Pathways prioritized
based on risk after
initial random samples
help develop a first
level of agricultural risk
Conduct additional
AQIM inspections
through statistical
random sampling
OSHA collects Injury
and Illness Logs from
all establishments in
the targeted population
Establishments are
required to keep Injury
and Illness Logs by law
Establishments with
Injury/Illness rates 2 to
3 times the national
average receive letters
encouraging them to
use OSHA's free
compliance assistance
services (approximately
13,000-14,000
establishments)
Inspections conducted
at 175 randomly
selected establish-
ments with low rates in
industries with high
rates
Inspections conducted
at a random number of
establishments that do
not submit the required
Injury and Illness Log
(continued)
WHD analyzes information
from directed and
complaint-driven
investigations to identify
potential areas of
noncompliance
Statistically valid
investigation-based
surveys help identify
noncompliance in
populations that may not
complain when there are
violations
National survey findings
show that particular
sources of noncompliance
may be more prevalent in
some geographic
locations
Surveys help develop
future targeting strategies
FMCSA ranks companies
according to prior 30-
month compliance history
and severity of data
SafeStat, a computer
algorithm, uses the
compliance data collected
by Federal and State
agents and the
crash/inspection data
State agencies collect to
create compliance
percentile rankings of
trucking companies
SafeStat stratifies
companies according to
safety records; the
companies most likely to
have crashes are selected
for inspections and
compliance reviews
Category A and Category
B are the highest priorities
(represent high risk);
these companies become
priorities for full onsite
compliance reviews and
inspections
(continued)
SAMHSA requires each
State to prepare its own
sampling plan that must
meet 10 standards/
components before
SAMHSA approves it
States can have
separate plans to
inspect known violators
in addition to the
SAMHSA required
random inspections
-------
ro
to
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
How
Establishments
are Selected for
Inspection
(continued)
Inspections conducted
at those establishments
with a history of the
most severe safety and
health violations
The Inspection Selection
System (ISS-2) uses
SafeStat results to
generate red, yellow, or
green light
recommendations to
assist States in selecting
trucks for roadside
inspections
Companies with no data in
the SafeStat database
receive a high risk rating,
"red," rating from the ISS-
2 that recommends State
agents conduct a roadside
inspection
Collaboration
Piggy-backed on a
former Customs
Service contract,
utilizing the contracted
statistician from
George Mason
University to review
early sampling
protocols
Assistance from US DA
statisticians at the
National Agricultural
Statistics Service in
developing and
reviewing the
methodology
Used external
statisticians for
developing the
sampling frames for
each pathway
OS HA works
collaboratively with
States to develop
annual and multi-year
strategic plans
Works with BLS for
data collection and
annual estimation of
rates
OSHA annually
receives OMB approval
to collect site specific
information on injuries
and accidents from
establishments
WHD works with
approximately 750
investigators from five
regions and 48 district
offices
Federal investigators
conduct the investigation-
based statistical surveys
External statisticians
develop a statistical
methodology, draw a
representative sample,
and conduct in-depth
analysis of statistically
valid surveys
FMCSA coordinates with
State licensing agencies,
who can require a U.S.
DOT registration number
from a company before
licensing its trucks
(Approximately half the
States do this now)
States conduct the
majority of roadside
inspections conducted a
year
John A. Volpe Center
National Transportation
Systems Center (Volpe
Center), a DOT
independent research
center, was the primary
contractor for the SafeStat
system
Consulted with external
statisticians to develop
the statistical sampling
frame
SAMHSA provided
guidance documents
and training materials
to States, community
groups, and merchants
SAMHSA collaborates
with community groups
and parents by
providing training to be
alert and report
violators
-------
ro
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
Benchmarking
APHIS focuses on
analyzing the reason(s)
for the differences
between two rates:
- Actual rate that
pests and diseases
enter through a
pathway, from
regularly scheduled
inspections
- Predicted rate, from
additional AQIM
random inspections
OSHA benchmarks the
Federal States'
progress against
Federal OSHA's
progress
OSHA tracks a core
group of 10,000-11,000
establishments to
measure injury and
illness rates over time
Uses national
investigation-based
statistical surveys to
monitor national, regional,
and local progress with
labor regulations
WHD prepared baselines
for targeted industries,
and develops trends to
show changes in
compliance
Goal to reduce large
truck-related fatality rate
by 41 % from 1996 to
2008, resulting in a 2008
rate of 1.65 fatalities per
100 million truck vehicle
miles traveled
Publishes rates on its
Website, where the public,
industry, and insurance
companies can access
performance information
The goal of the
program is to ensure
nation-wide
noncompliance no
higher than 20% by
2000, from a baseline
national rate of 42%
noncompliance in 1996
SAMHSA monitors a
State's performance
against the State's own
benchmark
Other Inspection
Program
Components
AQIM is an
inspectional process
conducted in addition
to regular inspection
activities
APHIS conducted
AQIM activities with
available resources
Identifying agricultural
risks is statistically
valid rather than risk
known from previous
experience
APHIS can objectively
target the pathways
posing the highest risk
to agriculture; may
include previously
unknown risks
Develops rates
quarterly
Five national emphasis
programs: exposure to
lead, exposure to silica,
amputations, trenching
in construction, and the
ship-breaking industry
Approximately 145
local emphasis
programs
Initially identified three
industries and used
national surveys to
develop a compliance
baseline: agriculture, the
garment industry, and
health care industry
The statistically valid
surveys supplement
targeted enforcement
programs
Uses various sources to
identify its universe: Dun
& Bradstreet and State
licensing agencies are
used most frequently
Maintained and used
compliance rates
internally since the 1980s
Publishes monthly
updates of percentile
compliance rates on its
Website
Additional weight is not
added for the size of the
company; an internal
process adds "a little more
weight" for companies that
haul hazardous waste and
passengers
Roadside inspection data
are very accurate because
of conditions in the Motor
Carrier Safety Assistance
Program grant
agreements
FMCSA provided
approximately $185
million in 2005 to States to
conduct roadside
inspections
SAMHSA also requires
States to annually input
the coded inspection
data into SAMHSA's
statistical package
From 1996 to 2000, the
national average of
retailer non-compliance
decreased from 42% to
approximately 19%
For 2006, the national
weighted non-
compliance rate was
10.9%
SAMHSA annually
publishes
noncompliance rates
for each State in its
Website
Statutory/
Regulatory
Authority to
Collect Data
Supports the mission
to mitigate and reduce
agricultural risk
entering into the United
States
Occupational Safety
and Health Act of 1970
Title 29 Code of
Federal Regulations
(CFR) 1904, Recording
and Reporting
Occupational Injuries
and Illnesses
Title 29 CFR 516,
Records to be Kept by
Employers (under the Fair
Labor Standards Act)
Title 49 CFR 350,
Commercial Motor Carrier
Safety Assistance
Program
Data submission part of
grants agreements
Section 1926 of Public
Health Service Act
(Synar Amendment)
Title 45 CFR 96,
Tobacco Regulation for
the Substance Abuse
Prevention and
Treatment Block Grant
-------
ro
cn
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHS SAMHSA
Analyses
Uses data to identify
the pathways that pose
the highest risk
Uses data to verify that
previously identified
pathways posing risks
to agriculture are still
valid
OSHA analyzes the
data from BLS
database
Collects information
from sites
Uses statistical
samples to select sites
for inspections
Used external
consultants for
evaluating results of
Site specific targeting
methodology
Federal investigators
conduct surveys
External statisticians
develop the statistical
methodology, draw a
representative sample,
and conduct in depth
analysis of statistically
valid surveys
Independent analyses by:
- University of
Tennessee
- Mathematica
- Boston University
Volpe Research Center
analyzes data and
conducts studies to
assess the success of the
compliance strategy
SAMHSA reviews the
inspection and
compliance data States
report. They analyze
data and investigate the
reasons for
discrepancies or
suspected problems
with data accuracy
Benefits
Evaluate intermediate
results and improve the
resources allocated to
pathways
Identify high risk
pathways that require
additional resources
Identify unknown risks
to agricultural
resources
APHIS can plan for
situations of current
funding and reduced
funding
OSHA can quantify its
program results, e.g., a
13% decrease in injury
rates when OSHA used
a combination of letters
and inspections
Can objectively
demonstrate the results of
enforcement and
compliance assistance
strategies
Can identify and pinpoint
types and areas of
noncompliance
Identify certain
populations of workers
who may not complain
when their employers are
noncompliant
Provide unbiased
assessment of compliance
FMCSA provides access
to current safety data and
crash statistics monthly on
its Website
FMCSA uses SafeStat
results to establish
Federal priorities for on-
site compliance reviews
Because FMCSA
publishes compliance
rates on its Website, and
updates them monthly,
companies have to stay
diligent
Compliance Review
Assessment Model and
Compliance Review
Effectiveness Study
SAMHSA can quantify
results of enforcement
strategies
From 1996 to 2000,
the national weighted
average of retailer non-
compliance decreased
from 42% to
approximately 19%
For 2006, the national
weighted non-
compliance rate is
10.9%
External
Reviews
A contracted
statistician reviewed
the AQIM
methodology; says the
concept is good
USDAOIG
OSHA (in general)
reviewed by GAO four
to five times a year
DOL OIG once or twice
Received all green
ratings from OMB's
Program Assessment
Rating Tool
OMB's Program
Assessment Rating Tool
reviewed WHD's
methodologies to ensure
valid and dependable
results
OMB was concerned that
WHD only focused on
three industries; WHD has
since expanded their
methods into other
industry areas
DOT OIG
GAO
Volpe Center
HHS OIG (once shortly
before and once shortly
after the Synar
amendment was
implemented)
GAO in 2001
-------
US DA APHIS
DOL OS HA
DOL ESA WHD
DOT FMCSA
HHSSAMHSA
Lessons
Learned
Need for a champion in
top management that
accepts/supports the
activity
Need for local
managerial support
(day-to-day champion)
If another agency
becomes involved in
the process, learn its
culture and adapt
Data quality is vital
The planning is never
entirely complete
Documentation of the
process/steps is key so
the methodology is
defensible to outsiders
Private companies may
be able to help with
logic models and
outcome measurement
activities
Random sampling
methodologies were
verified with different
levels of staff to assess
ability for complete
inspections
EPA's media programs
sound like "pathways"
Top management can
help obtain State data
Compliance assistance
combined with
inspections was more
effective in reducing
noncompliance.
Establishments were
not asking for
assistance when OSHA
decreased the number
of inspections in 1994-
1995
For maximizing its
impact, OSHA enters
into settlement
agreements with
companies that have
more than one
establishment in
violation
WHD secured support
from top leadership and
demonstrated
improvements in
compliance, to address
initial internal resistance
to using random samples
Using an external third
party for sampling and
analysis can remove the
perception of bias
Set realistic goals to
achieve compliance; e.g.,
WHD learned that annual
industry surveys do not
allow time to determine
the full impact of
intervention, and
improvement. WHD
changed to doing surveys
at 5-year intervals
Causes for violations are
different in different
industries. The same
approach is not effective
for everyone
Recognize different levels
of noncompliance, based
on number of affected
workers and the extent of
the noncompliance
Work collaboratively with
States to collect data
Require data and statistics
from States as part of
grant agreements
Accountability and
accuracy of data
increased because
external stakeholders use
data, e.g., industry,
insurance companies, and
the public
FMCSA used statistical
methods to maximize the
limited resources to bring
into compliance
companies most likely to
be involved in future
crashes
Fourteen local studies
showed that direct
enforcement activities
with penalties
drastically reduce
retailer noncompliance
Perception of
enforcement activities
is a big component of
the Synar program
Analyze data and verify
results to check that
they are realistic and
accurate. Analyze the
data and adjust
strategy based on the
cause of the problem
Source: OIG analysis of interviews with, and documents from, APHIS, OSHA, WHD, FMCSA, and SAMHSA
-------
Appendix C
Selected Sources
United States Department of Agriculture,
Animal and Plant Health Inspection Service, Plant Protection and Quarantine
USDA APHIS. 2006. Plant Protection and Quarantine Home Page. Available at:
http://www.aphis.usda.gov/ppq/
USDA. 2002. Agricultural Quarantine Inspection Monitoring (AQIM) Handbook, 10/2006-02
Edition. Marketing and Regulatory Programs, Animal and Plant Health Inspection Service, Plant
Protection and Quarantine. Available at:
http://www.aphis.usda.gov/ppq/manuals/port/AQIM Chapters.htm
USDA APHIS. 2005. Plant Protection and Quarantine Strategic Plan FY 2006-2009.
Available at: http://www.aphis.usda.gov/ppq/strategic-plan.html
USDA OIG. 2005. Animal and Plant Health Inspection Service: Transition and Coordination of
Border Inspection Activities Between USDA andDHS. Report No. 33601-0005-Ch. Available
at: http://www.usda.gov/oig/webdocs/33601-05-CH.pdf
Department of Labor,
Occupational Safety and Health Administration
OSHA. 2006. OSHA Home Page. Available at: http://www.osha.gov/
ERG. 2004. Evaluation of OSHA's Impact on Workplace Injuries and Illnesses in
Manufacturing Using Establishment-Specific Targeting of Interventions. ERG: Lexington, MA.
OSHA. 2005. OSHA Strategic Management Plan 2003-2008. Available at:
http://vvvvvv.osha.gov/StratPlanPublic/index.html
Department of Labor,
Employment Standards Administration, Wage and Hour Division
WHD. 2006. WHD Home Page. Available at: http://www.dol.gov/esa/whd/
DOL. 2006. U.S. Department of Labor Strategic Plan FY 2006-2011. Available at:
http://vvvvvv.dol.gov/ sec/stratplan/main.htm
WHD. 2005. 2005 Statistics Fact Sheet: Wage and Hour Collects $166 Million in Back Wages
for 241,000 Employees in FY 2005. Available at:
http://vvvvvv.dol.gov/esa/vvhd/statistics/20Q531 .htm
27
-------
Department of Transportation,
Federal Motor Carrier Safety Administration
FMCSA. 2006. FMCSA Home Page. Available at: http://www.fmcsa.dot.gov/
FMCSA. 2005. FMCSA Safety Program Effectiveness Measurement: Compliance Review
Effectiveness ModelResults for Carriers with Compliance Reviews in 2002. FMCSA-RI-05-
036. Available at:
http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/CREMMav20Q5.pdf
FMCSA. 2005. Research and Technology 5-Year Strategic Plan (2005-2009). Available at:
http://vvvvvv.fmcsa.dot.gov/facts-research/research-technologv/rt-5vear-strategicplan.pdf
FMCSA. 2006. Summary of Program Measures Activity. Available at:
http://ai.fmcsa.dot.gov/ProgramMeasures/Intro/ProgramMeasuresMain.asp
FMCSA. 2006. FMCSA 2006-2011 Strategic Plan, Draft. Available at:
http://vvvvvv.fmcsa.dot.gov/about/vvhat-vve-do/strategv/strategic-plan.htm
GAO. 2005. Large Truck Safety: Federal Enforcement Efforts Have Been Stronger Since 2000,
but Oversight of State Grants Needs Improvement. GAO-06-156. Available at:
http://vvvvvv.gao.gov/nevv.items/d06156.pdf
John A. Volpe Center. 2006. Measuring the FMCSA's Safety Objectives from March 2000 to
September 2004. FMCSA-RI-06-038. Prepared for FMCSA. Available at:
http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/StrategicPlan 2006.pdf
Department of Health and Human Services,
Substance Abuse and Mental Health Administration
SAMHSA. 2006. Synar Amendment: Protecting the Nation's Youth from Nicotine Addiction.
HHS SAMHSA Center for Substance Abuse Prevention. Available at:
http://prevention.samhsa.gov/tobacco/default.aspx
SAMHSA. 2003. Synar Regulation: Sample Design Guidance. HHS SAMHSA Center for
Substance Abuse Prevention.
SAMHSA. 2006. Guide for a Synar Sampling Frame Coverage Study. HHS SAMHSA Center
for Substance Abuse Prevention. Available at:
http://dovvnload.ncadi.samhsa.gov/csap/CSAPCoverageStudvGuide-2006.pdf
28
-------
Appendix D
Agency Response
May 22, 2007
MEMORANDUM
SUBJECT: The Office of Enforcement and Compliance Assurance's Agency Response to the
Draft Evaluation Report: Overcoming Obstacles to Measuring Compliance:
Practices in Selected Federal Agencies; Report No. 2006-00378
FROM: Granta Y. Nakayama (/s/ by Catherine McCabe for Granta Y. Nakayama)
TO: Jeffrey K. Harris
Director for Program Evaluation, Cross-Media Issues
Office of Inspector General
The purpose of this memorandum is to provide the Office of Enforcement and
Compliance Assurance (OECA) response to your April 5, 2007 memorandum which transmitted
the draft version of the evaluation report entitled: Overcoming Obstacles to Measuring
Compliance: Practices in Selected Federal Agencies. We appreciate the opportunity to comment
on the draft evaluation report.
OECA recognizes the value of examining the measurement practices of other agencies in
order to learn about approaches that might be useful to EPA's national enforcement and
compliance assurance program. In fact, several years ago OECA outreach to other regulatory
agencies resulted in identification of an approach to developing statistically-valid compliance
rates used by the U.S. Customs Service. That approach provided a methodology that combined
targeted inspections and random inspections to produce statistically-valid compliance rates.
OECA retained the consultant used by the Customs Service to develop the methodology that
OECA continues to use to produce statistically-valid rates for specific segments of the regulated
universe.
While the current draft report identifies five other federal programs that may have
similarities to EPA's compliance programs, OECA is unable to fully assess, from the information
provided in the draft report, the applicability of these practices to OECA's needs and
requirements for producing statistically-valid compliance rates. As we have discussed, EPA's
national compliance and enforcement program is responsible, along with the states, for
maximizing compliance with 12 environmental statutes, 28 distinct programs under those
statutes, and dozens of regulatory requirements under those programs which apply in various
combinations to a universe containing millions of regulated entities.
While the draft report acknowledges many obstacles OECA faces in measuring
compliance across the regulated universe, we do take issue with the characterization of OECA
"as resistant to conducting random inspections." OECA has in fact made conscious management
29
-------
choices on how best to use enforcement resources. The current methodology used for producing
statistically-valid rates combines inspections targeted at facilities likely to be in violation with
inspections conducted at randomly-selected facilities to produce a representative sample of the
regulated population. Because inspection resources are finite, every random inspection
conducted means sacrificing a targeted inspection likely to identify violations. This, OECA's
challenge has and will continue to be to carefully and effectively balance its measurement
approaches with its mission of protecting public health and the environment.
OECA acknowledges the information provided in the draft report and accepts the report
recommendations.
Recommendation 1 directs the Assistant Administrator for OECA to "establish a plan of
action with milestones to incorporate using statistical methods to demonstrate the results of
EPA's enforcement." OECA has previously accepted this recommendation.
In OECA's July 31, 2006 report to OMB entitled Expanding the Use of Outcome
Measurement for EPA's Office of Enforcement and Compliance Assurance, OECA made a
commitment to expand the use of statistically-valid compliance rates for specific noncompliance
patterns focused on national priorities or other important problem areas. This commitment will
entail development of an action plan.
The plan can be completed after two distinct but related reviews already underway are
completed by OECA. The first of these is a review of the strategies for the national priorities
chosen for the FY2008-2010 planning cycle. This review is designed to make the strategies
more performance-based by establishing clear goals and performance measures to guide the
management and oversight of the national priorities. The second review is designed to revise the
current compliance objective and sub-objectives in the EPA Strategic Plan so they are focused on
national priorities and problem areas, rather than their current orientation toward program tools
such as assistance, incentives, inspections, and enforcement. A commitment to make these
revisions was made as part of OECA's July 2006 report to OMB. This plan of action will be
completed by the end of calendar year 2007.
Recommendation 2 directs the Assistant Administrator for OECA to "coordinate with the
in-house statistical expertise available in EPA's Office of Research and Development and Office
of Environmental Information to help develop statistical models and evaluate external
proposals." OECA accepts this recommendation.
OECA has in place a contract and funding for statistical support that we believe will be
sufficient to perform the primary tasks associated with expanding the use of statistically-valid
compliance rates. However, we will examine the level of statistical expertise and the types of
statistical services that could be provided by staff in the Office of Research and Development
and the Office of Environmental Information in order to support our efforts to develop
meaningful compliance rates.
Thank you for the opportunity to comment. If you have any questions, you may contact
OECA's Audit Follow-up Coordinator, Gwendolyn Spriggs, on 202 564-2439.
30
-------
Appendix E
Distribution
Office of the Administrator
Office of General Counsel
Assistant Administrator for Enforcement and Compliance Assurance
Assistant Administrator for Environmental Information
Assistant Administrator for Research and Development
Agency Followup Official (CFO)
Agency Followup Coordinator
Associate Administrator for Congressional and Intergovernmental Affairs
Associate Administrator for Public Affairs
Senior Advisor to the Assistant Administrator for Enforcement and Compliance Assurance
Audit Followup Coordinator, Office of Enforcement and Compliance Assurance
Audit Followup Coordinator, Office of Environmental Information
Audit Followup Coordinator, Office of Research and Development
Acting Inspector General
31
------- |