* o \
I	o U.S. ENVIRONMENTAL PROTECTION AGENCY
% V-<- / OFFICE OF INSPECTOR GENERAL
PRO"^
Catalyst for Improving the Environment
Briefing Report
EPA Could Improve
RCRAInfo Data Quality and
System Development
Report No. 11-P-0096
February 7, 2011

-------
Abbreviations
EPA	U.S. Environmental Protection Agency
OIG	Office of Inspector General
ORCR	Office of Resource Conservation and Recovery
RCRAInfo Resource Conservation and Recovery Act Information System

-------
K U.S. Environmental Protection Agency	ii-p-oo96
9	\ Office of Inspector General	February 7,2011
I fi

At a Glance
Why We Did This Review
We sought to determine
whether data within the
Resource Conservation and
Recovery Act Information
System (RCRAInfo) complied
with prescribed system edit
and validation checks
designed to control data entry.
The Office of Inspector
General contracted with
Williams, Adley & Company,
LLP, to conduct this review.
Background
RCRAInfo collects data from
states and private companies
to track the generation,
shipment, and receipt of
hazardous solid waste. This
data is reported to the public
in the Biennial Hazardous
Waste Report and through the
Envirofacts website. The EPA
Office of Resource
Conservation and Recovery,
Information Collection and
Analysis is responsible for
development and ongoing
support of RCRAInfo.
For further information, contact
our Office of Congressional,
Public Affairs and Management
at (202) 566-2391.
The full report is at:
www.epa.aov/oia/reports/2011/
20110207-11 -P-0096.pdf
Catalyst for Improving the Environment
EPA Could Improve RCRAInfo Data Quality and
System Development
What Williams, Adley & Company, LLP, Found
RCRAInfo data that tracks hazardous waste handlers and the shipment and receipt
of hazardous waste contains errors and is missing source documentation. These
conditions call into question the quality and reliability of data within the
RCRAInfo system, as well as any resulting reporting. Further, RCRAInfo system
owners did not follow the prescribed System Life Cycle Management testing
procedures to test and validate the updated software and updated system. For
example, the edit and validation checks did not work as intended, the data
dictionary was not maintained, and system coding errors existed in the system.
Further, field work found instances of test data comingled with production data.
Overall, the above conditions were caused by not having specific data quality
procedures for RCRAInfo that align with the Agency's data quality policy, not
following the System Life Cycle Management procedures for system
development, and not adequately communicating with the states regarding the
RCRAInfo test environment. EPA is responsible for maintaining the technology
platform in a manner that supports data quality and integrity, as well as for
oversight of the data quality program. As required by the EPA quality policy, EPA
organizations must document and implement a quality program for environmental
data that is intended for external distribution.
What Williams, Adley & Company, LLP, Recommends
Williams, Adley & Company, LLP, recommends the Director, Office of Resource
Conservation and Recovery, Office of Solid Waste and Emergency Response,
implement a procedure for regional personnel to notify a state when changes are
made to handler records. The company also recommends that guidance and policy
be provided on retaining source documentation, and that control procedures be
implemented on updating documentation and on reviewing the production
database for test data on a semiannual basis.
We met with Agency officials to discuss our findings and recommendations.
The Agency provided responses, and we included those comments in the briefing
report.

-------
* o \
I® I
V PRO^
THE INSPECTOR GENERAL
February 7, 2011
MEMORANDUM
SUBJECT:	EPA Could Improve RCRAInfo Data Quality and System Development
Report No. ll-P-0096
FROM:	Arthur A. Elkins, Jr.
Inspector General
TO:	Suzanne Rudzinski, Acting Director
Office of Resource Conservation and Recovery
Office of Solid Waste and Emergency Response
Attached is the briefing report on the subject audit conducted by Williams, Adley &
Company, LLP (Williams Adley), on behalf of the Office of Inspector General (OIG) of
the U.S. Environmental Protection Agency (EPA). This report contains findings that
describe the problems Williams Adley identified and corrective actions recommended.
This report represents the conclusions of Williams Adley and does not necessarily
represent the final EPA position. Final determinations on matters in this report will be
made by EPA managers in accordance with established audit resolution procedures.
The estimated cost for performing this audit, which includes contract costs and OIG contract
management oversight, is $855,110.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to
this report within 90 calendar days. You should include a corrective actions plan for
agreed-upon actions, including milestone dates. Your response will be posted on the
OIG's public website, along with our memorandum commenting on your response. Your
response should be provided as an Adobe PDF file that complies with the accessibility
requirements of section 508 of the Rehabilitation Act of 1973, as amended. The final
response should not contain data that you do not want to be released to the public; if your
response contains such data, you should identify the data for redaction or removal. We
have no objections to the further release of this report to the public. This report will be
available at http://www.epa.gov/oig.
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460

-------
If you or your staff have any questions regarding this report, please contact Patricia H.
Hill at 202-566-0894 or hill.patricia@epa.gov. or Rudolph M. Brevard at 202-566-0893
or brevard.rudv@epa.gov.

-------
EPA Could Improve RCRAInfo
Data Quality and
Svstem Developnletit
Results of Review
11-P-0096
1

-------
Audit Methodology
Documented the information flows and system controls over the
Resource Conservation and Recovery Act Information System
(RCRAInfo).
Did testing from January 2007 through December 2008.
Conducted data integrity testing to evaluate compliance with
RCRAInfo business rules (data dictionary and validation checks).
Conducted data quality testing in five locations (Virginia, California,
Arizona, Hawaii, and Nevada) to verify accuracy of data entered into
RCRAInfo.
Tested 45 EPA handler IDs per state for Handler and Biennial Report
data.
Tested 25 EPA IDs for manifest data (5 manifests per state).
11-P-0096
2

-------
Data Quality Testing of Handler Data
Exceptions
15%
Unavailable
30%
Correct
55%
Forty-five percent of the handler data values in RCRAInfo either did not match the source
documentation (data entry forms) or were not available for review.
11-P-0096
3

-------
Reasons for Data Quality Errors and
Unavailable Supporting Documentation
State environmental agency staff stated that data is sometimes
altered at the regional level and the state environmental office is not
always notified.
State environmental staff stated that updates to RCRAInfo data is
sometimes initiated by phone calls from generators or handlers due
to emergencies at the facility or other time sensitive events. State
environmental staff sometimes update RCRAInfo data but do not
always update hardcopy files.
EPA does not have a consistent policy for states to retain source
documentation.
State environmental staff do not consistently apply the established
methodology for converting unit/weight/volume amounts on
manifests for entry into RCRAInfo.
11-P-0096
4

-------
OIG Recommendations
Director, Office of Resource Conservation and Recovery (ORCR),
Office of Solid Waste and Emergency Response, should:
1	Implement a procedure for regional personnel to notify a state
when changes are made to handler records.
2	Conduct training for data entry personnel regarding the proper
procedures for documenting updates to RCRAInfo.
3	Provide guidance and policy from the program level regarding
retaining source documentation.
4	Provide additional guidance to states to facilitate accurate
conversion of hazardous waste amounts as required for input
into RCRAInfo.
11-P-0096
5

-------
the Fi ndings
ORCR stated that during the 2009 biennial report cycle, the
states, not the regions, loaded the biennial report data directly
into RCRAInfo.
ORCR stated that it planned and budgeted for increases in
state/regional training for years where there is no national
conference. The importance of this training will be stressed
during the conferences.
ORCR stated that the conversion of weights and measures will
be stressed during the national conferences and expanded
training.
ORCR stated that it will stress the importance of identifying
critical documents needed to support program needs and
establish retention guidelines.
11-P-0096
6

-------
RCRAInfo System Development Practices
Need Improvement
The data dictionary was not updated to reflect changes in
RCRAInfo business rules.
System code errors existed in the application software.
Test data existed in the production database.
The test environment was not used consistently.
11-P-0096
7

-------
Reasons For System Life Cycle Management
Procedure
EPA personnel did not ensure that RCRAInfo version 4 system
documentation accurately depicted the system's actual design.
Svstem testina did not detect svstem code errors in RCRAInfo
The program office did not enforce use of test environment for
testing purposes.
No formal process is in place to identify test data in the production
database.
11-P-0096
8

-------
OIG Recommendations
Director, Office of Resource Conservation and Recovery (ORCR),
Office of Solid Waste and Emergency Response, should:
5 Implement a control procedure to enforce that business rule
changes are updated in the RCRAInfo system documentation.
e Implement a control procedure to enforce that RCRAInfo coding
changes are tested prior to placement into production.
7 Notify states to use the test environment for all testingjnsteac|
of the production environment.
s Implement a procedure to review the production database for
test data on a semiannual basis.
11-P-0096
9

-------
EPA's Responses to the Findings
IT	e3
ORCR stated it currently has a process in place and has been
working with states to correct orphan data. Software will be
developed as each module is released and will be run monthly to
detect orphan data.
ORCR stated that application software does go through extensive
testing by internal developers, independent contractors, and the
user community before it is released into production.
ORCR stated that it stresses to the user community that only
preproduction data should be used for testing or data entry
training. ORCR will again stress the importance of using the test
environment for test data.
ORCR stated that it is in the process of developing a tool for the
purpose of verifying test data. Once developed, this tool will be run
monthly to identify any test data in production.
11-P-0096
10

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Claimed
Amount
Agreed To
Amount
Implement a procedure for regional personnel to
notify a state when changes are made to handler
records.
5 Conduct training for data entry personnel regarding
the proper procedures for documenting updates to
RCRAInfo.
5 Provide guidance and policy from the program level
regarding retaining source documentation.
Provide additional guidance to states to facilitate
accurate conversion of hazardous waste amounts
as required for input into RCRAInfo.
Implement a control procedure to enforce that
business rule changes are updated in the
RCRAInfo system documentation.
Implement a control procedure to enforce that
RCRAInfo coding changes are tested prior to
placement into production.
Notify states to use the test environment for all
testing instead of the production environment.
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
Implement a procedure to review the production
database for test data on a semiannual basis.
Director, Office of Resource
Conservation and Recovery,
Office of Solid Waste and
Emergency Response
1 0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
11-P-0096
11

-------
Appendix A
Distribution
Office of the Administrator
Assistant Administrator for Solid Waste and Emergency Response
Acting Director, Office of Resource Conservation and Recovery, Office of Solid Waste and
Emergency Response
Agency Followup Official (the CFO)
Agency Followup Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Affairs
Associate Administrator for External Affairs and Environmental Information
Audit Followup Coordinator, Office of Solid Waste and Emergency Response
11-P-0096
12

-------