$
<
33
\
mj
¦p.. ,\s
PRO'S4-
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Report to the Acting Inspector General
Special Project: Measuring the
Quality of OIG Reports
Report No. 2006-M-00015
September 19, 2006
-------
Report Contributors:
Robert K. Bronstrup
Patrick Gilbride
Office of Congressional and Public Liaison
Abbreviations
AIG
Assistant Inspector General
DCAA
Defense Contract Audit Agency
EPA
U.S. Environmental Protection Agency
GAGAS
Generally Accepted Government Auditing Standards
GAO
Government Accountability Office
GAS
Government Auditing Standards (2003 Revision)
IGEL
Inspector General E-Learning
IGOR
Inspector General Operations and Reports System
OA
Office of Audit
OCPL
Office of Congressional and Public Liaison
OIG
Office of Inspector General
OMB
Office of Management and Budget
OPE
Office of Program Evaluation
QA
Quality Assurance
-------
^tos%
0VO
^ s UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
\ ^ I WASHINGTON, D.C. 20460
*l PRO"*^
OFFICE OF
INSPECTOR GENERAL
September 19, 2006
MEMORANDUM
SUBJECT: Special Project: Measuring the Quality of OIG Reports
Report No. 2006-M-00015
FROM: Robert K. Bronstrup /s/
Director, Central Resource Center
Patrick Gilbride /s/
Director, Western Resource Center
TO: Bill Roderick
Acting Inspector General
Attached is the final report of our special project on Measuring the Quality of Office of Inspector
General (OIG) reports. Specifically, we tested a process to score specific quality characteristics
of 26 OIG reports issued between October 1, 2005, and March 31, 2006. Also, we made
observations and recommendations to you that we believe will strengthen the audit, evaluation,
and liaison processes. The scoring form we used in our review is included as Appendix A. The
scoring form the Office of Congressional and Public Liaison used to assess draft reports is
included as Appendix B. We explain the specific attributes for which we reviewed OIG reports
in the Scope and Methodology section which is included as Appendix C.
We received comments from the Assistant Inspectors General (AIGs) and provided them to you.
We also provided to you and to the AIGs a summary of all of the comments received on the
report and the scorecards. We used their comments in preparing the final report and
recommendations and also in revising individual scorecards. If you have any questions about the
final report or our observations and recommendations, please contact Robert Bronstrup at
312-886-7169 or Patrick Gilbride at 303-312-6969.
cc: Acting Deputy Inspector General
-------
Special Report: Measuring the Quality of OIG Reports
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Summary of Results 1
Measuring the Quality of OIG Reports 1
Scoring the Results 2
2 Workpaper Enhancements 6
Documenting Supervisory Review of Workpapers 6
Cross-Referencing Assignment Guide Work Steps
to Supporting Workpapers 8
Enhancing the Independent Referencing Process 9
Approving Preliminary Research and Assignment Guides 11
3 Reporting Enhancements 12
Defining When Reports Should Use the Word "Official" 12
Ensuring Visual Aids Show Source of Data 12
Describing the Approach for Each Objective in
Scope and Methodology 13
4 Administrative Enhancements 15
Ensuring Staff Charge Time in a Uniform Manner 15
Entering Performance Measurement and Results System
(PMRS) Data 17
Appendices
A Project Quality Scorecard 18
B Report Quality Scoresheet for Draft Submissions 21
C Scope and Methodology 26
-------
Chapter 1
Introduction
Purpose
The purpose of this pilot project was to apply a set of experimental criteria to measure quality in
Office of Inspector General (OIG) reports issued by the Office of Audit (OA), Office of Program
Evaluation (OPE), and the Office of Congressional and Public Liaison (OCPL). Measuring the
quality of OIG work is important because it provides data that can be used to identify areas in
improving OIG processes. We developed quality measurement criteria and applied it to 26 EPA
OIG reports issued between October 1, 2005, and March 31, 2006.
We did not include single audit reports, Defense Contract Audit Agency (DCAA) contract audit
reports, or the Audit of EPA's Fiscal 2005 Financial Statements. Should single audits and the
audit of EPA's Financial Statements remain with the OIG, the scoring system will also include
those audits.
Summary of Results
To improve the quality of reports and work processes, the OIG should:
• Issue an interim policy to ensure supervisory reviewer notes are kept in a central location of
the workpapers.
• Enhance workpapers to better ensure (1) assignment guides are reviewed and approved,
(2) assignment guides are fully cross-referenced to the workpapers, and (3) the work of the
independent referencer is fully documented in the workpapers.
• Improve reports by ensuring reports (1) identify the specific titles of agency employees or
others who are cited in OIG reports, and use the term "official" for only SES-level
employees, (2) show the source of information for tables and charts, and (3) specifically
describe the methodology for addressing each objective.
• Develop a policy that will better ensure staff charge time to their direct assignments and to
indirect job codes in a more uniform manner in order to accurately determine the actual cost
of each assignment.
• Strengthen its followup process so that the final impact of our work can be better
determined.
Measuring the Quality of OIG Reports
The primary goal of OIG reporting is to keep the Administration and Congress fully informed of
issues impacting EPA programs as well as EPA's progress in taking action to correct those
issues. Other customers, based on their impact on our budget, are the Government
Accountability Office (GAO) and the Office of Management and Budget (OMB). In developing
our criteria to measure quality, we know that these customers view timeliness of our products as
1
-------
very important; therefore, timeliness is a high quality characteristic. Likewise, compliance with
the Generally Accepted Government Auditing Standards (GAGAS), found in Government
Auditing Standards (GAS), is required, and thus, is a high quality characteristic. Further,
potential cost savings, improving policy, or the environment are other important quality
characteristics for an organization that is a "catalyst for change."
With that in mind, the OIG should strive to consistently provide products that meet specific
quality characteristics and adhere to all applicable standards and OIG policies and procedures.
Accordingly, a measuring process should provide a mechanism to evaluate individual products
against specific quality criteria. The measuring process should also present the information in a
manner that, over time, will allow the OIG to assess trends in quality so that necessary
adjustments can be made to policies, procedures, and activities. The criteria used in this project
to assess quality in OIG reports were:
• Project cost.
• Documentary reliability of evidence.
• Supervision.
• Timeliness in preparing draft reports.
• Readability of reports, including whether the reports are clear, concise, convincing,
logical, and relevant.
A scoring form to measure and score these characteristics provides the organization with a
measurement of product quality and also serves as a basis for measuring a manager's
performance. The specific manner in which we calculated points is shown in our project quality
scorecard in Appendix A. The report quality scoresheet the OCPL Publications Unit uses to
score draft reports is in Appendix B.
Scoring the Results
The total quality scores, as well as the timeframes and project costs for the 26 OIG reports, are
shown in Table 1. Each total quality score is the sum of the two scoring systems: one for project
quality characteristics (Table 2) and the second for report quality characteristics (Table 3). Two
of the project quality characteristics we did not score. As noted in our observations, supervisory
review notes are not maintained consistently in the workpapers. As a result, we did not score
Supervision for the reports issued. Also, we did not score Significance, because the full impact
for certain reports could not be determined since the reports had just been issued. An Inspector
General Statement will be issued by October 1, 2006, that fully explains the scoring process and
all the criteria in both scoresheets.
In addition to scoring reports using the quality criteria, we are also are providing observations on
specific aspects of OIG work activities and processes that can be improved. Our observations
fall into three categories: (1) workpaper enhancements, (2) reporting enhancements, and (3)
administrative enhancements.
2
-------
Table 1. Total Quality Scores
Report No.
Staff
Days
Project
Cost
($000s)
Elapsed
Days
(Kickoff to
OCPL
Reviewing
Draft)
Elapsed
Days
(Kickoff
to Final
Report
Date)
Total
Project
Score
(Tbl.2)
Total
Weighted
Report
Score
(Tbl. 3)
Total
Quality
Score
2006-P-00001
387
302
181
237
6.0
5.0
11.0
2006-P-00002
832
694
168
214
6.0
4.7
10.7
2006-P-00003
508
397
533
651
1.0
4.5
5.5
2006-P-00004
96
80
143
232
6.0
5.5
11.5
2006-P-00006
567
443
374
471
3.0
4.6
7.6
2006-P-00007
453
367
410
536
4.0
7.9
11.9
2006-P-00008
209
169
361
422
5.0
8.4
13.4
2006-P-00009
558
436
413
544
3.0
5.2
8.2
2006-P-00010
See P-2
See P-2
260
320
5.0
5.2
10.2
2006-P-00011
428
334
98
125
3.0
7.0
10.0
2006-P-00012
103
86
120
161
7.0
7.9
14.9
2006-P-00013
1,126
879
279
511
4.0
4.3
8.3
2006-P-00014
See P-11
See P-11
87
146
3.0
8.4
11.4
2006-P-00015
663
553
377
495
5.0
7.2
12.2
2006-P-00016
896
700
400
546
4.0
4.9
8.9
2006-P-00017
631
493
319
440
3.0
5.5
8.5
2006-P-00018
240
187
90
145
6.0
7.3
13.3
2006-P-00019
See P-2
See P-2
320
376
5.0
5.0
10.0
2006-P-00020
See P-2
See P-2
329
377
5.0
4.4
9.4
2006-P-00021
See P-2
See P-2
320
378
5.0
6.1
11.1
2006-M-000004
284
222
199 -NO GO
N/A
N/A
9.2
9.2
2006-01-00018
231
193
512
589
0.0
8.3
8.3
2006-01-00021
228
190
199
213
5.0
9.7
14.7
2006-01-00024
See 00018
See 00018
542
644
0.0
9.4
9.4
2006-4-00026
49
41
93
105
7.0
8.6
15.6
2006-4-00027
38
32
61
67
7.0
9.6
16.6
3
-------
Table 2. Project Quality Scorecard
Report
Number
Evidence
Rating
Report
Timeliness
Deduction
Prelim.
Res
Guide
Fieldwork
Guide
Finding
Outlines
Total
Project
Score
2006-P-00001
4
0
N/A*
1
1
6.0
2006-P-00002
4
0
N/A
1
1
6.0
2006-P-00003
4
-6
1
1
1
1.0
2006-P-00004
4
0
0
1
1
6.0
2006-P-00006
4
-3
1
1
3.0
2006-P-00007
3
-1
0
1
1
4.0
2006-P-00008
4
-2
1
1
1
5.0
2006-P-00009
3
-3
1
1
1
3.0
2006-P-00010
4
-1
N/A
1
1
5.0
2006-P-00011
3
0
0
3.0
2006-P-00012
4
0
1
1
1
7.0
2006-P-00013
4
-1
N/A
1
4.0
2006-P-00014
3
0
0
3.0
2006-P-00015
4
-1
1
1
5.0
2006-P-00016
4
-3
1
1
1
4.0
2006-P-00017
4
-2
0
1
3.0
2006-P-00018
4
0
N/A
1
1
6.0
2006-P-00019
4
-1
N/A
1
1
5.0
2006-P-00020
4
-1
N/A
1
1
5.0
2006-P-00021
4
-1
N/A
1
1
5.0
2006-M-000004
N/A
N/A
0
0
N/A
2006-01-00018
4
-5
0
1
0
0.0
2006-01-00021
4
0
0
1
0
5.0
2006-01-00024
4
-5
0
1
0
0.0
2006-4-00026
4
0
1
1
1
7.0
2006-4-00027
4
0
1
1
1
7.0
Note: In certain assignments a preliminary research guide was not necessary and N/A is shown.
-------
Table 3. Report Quality Scorecard
Report No.
Readability Index
Grade Level
Readability
Score
Complete,
Concise,
Clear
Report Score
Total Weighted
Report Score
(Report Score
Divided by 10)
2006-P-00001
16.9
1
49
50
5.0
2006-P-00002
23.4
0
47
47
4.7
2006-P-00003
16.4
6
39
45
4.5
2006-P-00004
16.4
6
49
55
5.5
2006-P-00006
16.9
1
45
46
4.6
2006-P-00007
10.2
30
49
79
7.9
2006-P-00008
10.7
30
54
84
8.4
2006-P-00009
15.4
16
36
52
5.2
2006-P-00010
16.8
2
50
52
5.2
2006-P-00011
15.3
17
53
70
7.0
2006-P-00012
13.6
30
49
79
7.9
2006-P-00013
16.6
4
39
43
4.3
2006-P-00014
14.5
25
59
84
8.4
2006-P-00015
14.9
21
51
72
7.2
2006-P-00016
17.4
0
49
49
4.9
2006-P-00017
16.7
3
52
55
5.5
2006-P-00018
14.9
21
52
73
7.3
2006-P-00019
18.7
0
50
50
5.0
2006-P-00020
18
0
44
44
4.4
2006-P-00021
15.7
13
48
61
6.1
2006-M-000004
14.3
27
65
92
9.2
2006-01-00018
14.8
22
61
83
8.3
2006-01-00024
13.5
30
64
94
9.4
2006-01-00021
12.9
30
67
97
9.7
2006-4-00026
14.8
22
64
86
8.6
2006-4-00027
14
30
66
96
9.6
2006-S-00001*
18.7
0
67
67
6.7
*Note that 2006-S-00001, Fiscal Year 2005 Status of EPA's Computer Security Program, is a synopsis of the results of the other
OIG FY 2005 information security audits. We did not score this report on the Project Scorecard and it is not included in Table 1 or
Table 2.
-------
Chapter 2
Workpaper Enhancements
Documenting Supervisory
Review of Workpapers
The workpapers for the 26 OIG reports had
some supervisory review notes or comments
located throughout the workpapers.
However, the comments were not
maintained consistently and they were not
located in one section of the workpapers.
As a result, we could not score the reports
for "Supervision" using the scoring criteria.
We could not determine the frequency of
supervisory reviews in accordance with
proposed guidance provided to us at the beginning of our review by the Acting Inspector
General. The Acting Inspector General stated that supervisory reviews of workpapers prepared
by staff at or below the GS-9 level should occur at least twice a month and all other workpapers
should be reviewed at least once a month. Further, unless supervisory review comments are
maintained in one location of the workpapers, an independent referencer or peer review team
will have difficulty determining whether all supervisory review comments are resolved before
beginning independent referencing of the report.
The current 01G Project Management Handbook {Handbook) provides the following guidance
on supervisory review of workpapers:
All working papers must be reviewed by a member of the team who did not prepare the
working paper. Project or Assignment Managers' review of working papers will be
conducted to the extent necessary for the manager to ensure himself or herself that
working papers comply with standards. Working papers prepared by the Project or
Assignment Manager should be reviewed by an experienced team member or respective
Product Line Director. Evidence of working paper review must be recorded in the
working papers. (January 14, 2005, edition, p. 23)
The Handbook, however, does not require that supervisory review comments be kept in a central
location of the workpapers, nor does it state how frequently reviews should occur.
When supervisors prepare review comments, AutoAudit® does not capture those comments and
maintain them in a central location. AutoAudit® could be enhanced to more easily capture
supervisory review comments and to maintain them in one location of the workpapers. That step
would help independent referencers and peer reviewers determine whether all supervisory review
Evidence
§7.48 Sufficient, competent, and relevant evidence is
to be obtained to provide a reasonable basis for the
auditors' findings and conclusions.
§7.68 Audit documentation serves to (1) provide the
principal support for the auditors' report, (2) aid
auditors in conducting and supervising the audit, and
(3) allow for the review of audit quality. Audit
documentation should be appropriately detailed to
provide a clear understanding of its purpose and
source and the conclusions the auditors reached, and it
should be appropriately organized to provide a clear
link to the findings, conclusions, and recommendations
contained in the audit report.
GAS 2003 Revision
6
-------
comments were resolved. It would also allow a determination of the frequency of reviews
according to the guidance given by the Acting Inspector General. If workpaper reviews result in
no comments, the reviewer could state that the workpapers were complete and that no additional
work was needed.
To help strengthen the review of OIG workpapers the OIG should:
Recommendation 1: Issue an interim policy to clarify how to record and maintain
reviewer comments in a central location of the workpaper file for each assignment. The
reviewer comments must contain the dates of review, who performed the review,
disposition, and clearance of the response by the reviewer.
OIG Actions Taken and Planned: Inspector General Statement No. 1 was issued July
27, 2006, and provides direction in response to this recommendation.
All of the requirements of Inspector General Statement No. 1 should be incorporated into
the next revision of the Project Management Handbook.
Recommendation 2: Require that all workpapers be reviewed by the Project Manager
and that the Assignment Manager review all workpapers prepared by the Project Manager.
If the reviewer has no comments, the supervisor should add a short description such as "I
have reviewed the working papers and found them to be satisfactory."
OIG Actions Taken and Planned: Inspector General Statement No. 1 incorporates this
recommendation.
This recommendation does not prevent peer review of workpapers which is a means by
which team members can stay abreast of ongoing work. However, peer review of
workpapers does not constitute supervisory review.
Recommendation 3: Require the assignment manager and the product line director
review the workpapers that support the report.
OIG Actions Taken and Planned: The Handbook should be revised to incorporate this
recommendation. Until it is revised, each office should issue instructions to staff to ensure
this recommendation is timely implemented.
Product Line Directors' and Assignment Managers' review of the specific workpapers
supporting the report does not duplicate the responsibility of the independent referencer
whose responsibility is part of the OIG's quality assurance process. The Directors and
Assignment Managers have the specific responsibility as managers on the assignment to
perform reviews of the work performed and reviews of audit documentations supporting
the report. Implementing this recommendation will help to ensure that adequate evidence
supports each of the findings and recommendations in OIG reports.
7
-------
Recommendation 4: Require that all workpapers prepared by staff at the GS-9 or below
grade levels be reviewed no less than twice a month and all other workpapers should be
reviewed at least once a month.
OIG Actions Taken and Planned: This recommendation has been incorporated into
Inspector General Statement No. 1 and will be incorporated into the next revision of the
Handbook.
Recommendation 5: Require all reviewer comments to be resolved before the report is
submitted to the independent referencer.
OIG Actions Taken and Planned: This recommendation has been incorporated into
Inspector General Statement No. 1 and will be incorporated into the next revision to the
Handbook.
Cross-Referencing Assignment Guide Work
Steps to Supporting Workpapers
Evaluating assignment guides for the 26 reports reviewed showed 6 assignments for which not all
of the work steps were indexed back to the workpapers. The work steps may have been deleted as
unnecessary and the staff may have forgotten to note the rationale in the assignment guide. The
work steps may have been performed but the appropriate workpapers were not cross-referenced in
the assignment guide. In one instance, several work steps for a recent Katrina review were not
cross-referenced to workpapers in the assignment guide. When we contacted the Director, the
Director explained that the team decided the work was easier to track by EPA region rather than
by objective, which was how the assignment guide was initially set up. As a result, workpapers
were maintained by region and not linked to the objectives and work steps as stated in the
assignment guide. After notifying the Director, the team completed the assignment guide.
Not having all work steps cross-referenced to the workpapers, without some explanation, raises
concern as to whether the work necessary to complete the assignment was performed. The
Handbook states "Any steps omitted from the guide should be approved by the Assignment
Manager." Under "Field Work Conducted" is the statement, "As field work progresses, the team
continually updates finding outlines and maintains the Quality Assurance (QA) Checklist as
various field work activities are completed."
8
-------
Recommendation 6: Amend the Handbook to include the following:
a) Insert under "Field Work Conducted" language that requires the team to
continually update each assignment guide section with appropriate cross references
to the workpapers as work progresses;
b) Insert "Have the steps in the assignment guide been fully indexed to the supporting
workpapers or otherwise noted as to why the step has not been completed" into the
QA Checklist;
c) Insert, as part of the independent referencer's responsibilities, language into
Appendix 7 "Independent Referencing Guidance/Certification Memo" regarding
the need to determine if the guide has been fully indexed to supporting workpapers
(or reasons why the team did not complete steps) prior to undertaking referencing.
In those instances where the assignment guide is incomplete, the referencer will
notify the assignment manager and require completion before referencing begins.
OIG Actions Taken and Planned: The next revision to the Handbook will incorporate
this recommendation. Until the revision is issued each office should issue instructions to
ensure staff implement this recommendation.
Enhancing the Independent Referencing Process
The following areas in the OIG's independent referencing activity could be enhanced:
Grade Level and Independence of Staff Performing Independent Referencing
As noted in the Handbook, independent referencing should be assigned to experienced staff who
have knowledge of the Government Auditing Standards and OIG policies. Specifically, the
Handbook states:
Product Line Directors assign experienced staff to reference draft products. Product Line
Directors should select auditors/program analysts (usually a GS-12 or higher) for the
referencing assignment. The selected auditor or program analyst must possess a high
degree of independence, objectivity, experience, and knowledge of the Government
Auditing Standards and OIG reporting policies and standards.
However, under current OIG promotion guidelines, staff who are hired as GS-9s could be
promoted to GS-12s and then selected to independently reference reports with as little as 2 years'
experience. Also, staff from OA, OPE, and OCPL are assigned to independently reference
reports within their own product lines. Independent referencing should be performed by
individuals with several years of experience who are at a higher grade level and who are
independent of the product line. This will provide additional assurance that the referencers will
have the experience, knowledge, and independence necessary to carry out the independent
referencing. We noted that the Director of Assistance Agreements requires that a GS-14 perform
all of the independent referencing in that product line.
9
-------
Recommendation 7: The OIG should establish a centralized group of experienced
independent referencers separate from OA, OPE, OMS, and OCPL.
OIG Actions Taken and Planned: The OIG is establishing an Office of Quality
Assurance and Inspections which will have the responsibility for implementing this
recommendation.
Documenting the Independent Referencer's Work More Consistently
For the reports we reviewed, the independent referencers documented their comments, and
included their comment sheets in the workpapers. However, we noted that the independent
referencers do not always indicate review and acceptance of the supporting materials as required
by the Handbook. Specifically, Appendix 7, "Independent Referencing Guidance/Certification
Memo" in the Handbook states: "Use a colored pencil for placing tick marks on the document to
indicate verification and satisfaction with the supporting material. For example, place a tick
mark over each figure, date, citation to legal or other reference material, and proper name."
Further, for 11 of 26 reports reviewed, a copy of the indexed version of the report with the
independent referencer's tickmarks could not be found in the workpapers. As a result, there is
uncertainty as to whether each line of text of the indexed copy of the report was properly
referenced. We did not always see an affirmative statement by the independent referencer that
he or she believed the opinions and conclusions in the report were reasonable and consistent with
the facts presented and that the recommendations logically followed from the facts and
conclusions, as required by Appendix 7.
The independent referencer may have printed out a hard copy of the report that was indexed and
complied with the requirement in the referencing guidance, but he/she did not scan in or
otherwise ensure that the indexed version of the report with the referencer's tick marks was
placed in the workpapers. The hard copy of the indexed report the independent referencer used
may either still be with him or her or the Assignment Manager.
Recommendation 8: Require the indexed copy of the report with the tick-marks of the
independent referencer be kept in AutoAudit®. The independent referencer should include
a statement in his/her comments that the opinions and conclusions in the report are
reasonable and consistent with the facts presented and that recommendations logically
follow from these facts and conclusions.
OIG Actions Taken and Planned: The OIG will incorporate this recommendation into
the next revision of the Handbook Until the Handbook is revised, each office should
issue instructions to staff to ensure this recommendation is followed. In implementing this
recommendation, teams are not required to scan in the referencer's copy with colored
pencil tick marks. Teams can use a version of the report with electronic tickmarks.
During our review, we noted that teams had developed an electronic method for creating
and capturing tick-marks within AutoAudit® and this is acceptable.
10
-------
Approving Preliminary Research and Assignment Guides
With respect to the Assignment Guide for carrying out the fieldwork, the Handbook states:
In most cases, project guide changes may be approved by the Project Manager. Any
steps omitted from the guide should be approved by the Assignment Manager. The
guide is to be signed by the Product Line Director (or the Project or Assignment Manager
if delegated that authority by the Director). Significant changes to the guide must be
justified and approved in writing by the Director, in consultation with the applicable
Assistant Inspector General.
This paragraph in the Handbook is not clear as to what constitutes significant changes but that
most changes can be approved by the Project Manager (the GS-13 level). Also, the Handbook
does not state when the guide can be signed by the Product Line Director or delegated to the
Assignment Manager. Since OIG reviews can take hundreds of staff days, the assignment guide
is an important document to guide fieldwork. The Director should be responsible for signing the
assignment guide and the circumstances under which the project guide can be changed by the
Assignment/Project Manager should be clarified.
As a best practice, we observed in one assignment that the Product Line Director signed and
dated the front page of the Assignment Guide and then scanned it back into AutoAudit®. This
step showed the guide was in place before the entrance conference as required by the Handbook
Recommendation 9: Clarify the Handbook regarding what constitutes significant changes
and when approvals and subsequent changes to the assignment guide be made by the
Assignment Manager and Product Line Director.
OIG Actions Taken and Planned: The Handbook will be revised to clarify
responsibilities, and indicate that significant changes include dropping an objective or
deciding to implement a vastly different approach to accomplishing the objective.
Until the Handbook is revised, we recommend that each office issue instructions to staff to
ensure this recommendation is timely implemented.
11
-------
Chapter 3
Reporting Enhancements
Defining When Reports Should Use the Word "Official"
OIG reports do not always identify the title of the individual providing comments. Instead
reports use the word "official" even for lower level Agency personnel. When "official" is used
frequently in a report, the reader has difficulty judging the credibility of the comments.
A more reasonable approach is to use the titles of the Agency employees who are providing
comments in OIG reports. As stated in a U.S. Government Accountability Office Report Style
Manual.
In the body of the report we normally identify the official by title, making the comments
so that the reader of the report will be in a position to judge the credibility of the
comments.
The Handbook and the OIG Report Formatting and Style Guide do not specifically address this
issue.
Recommendation 10: The Handbook and Report Formatting and Style Guide should be
revised to reflect these following concepts and managers should ensure the guidance is
followed when preparing reports:
a. Use the word "official" to represent SES or higher level employees when the specific
title of the individual providing comments cannot be used in OIG reports.
b. For employees below the SES level, when reports cannot refer to their title, the
employee should be referred to as staff member of a specific office/division etc.
OIG Actions Taken and Planned: The OIG will incorporate this recommendation into
the next revision of the Handbook and the Report Formatting and Style Guide. Until it is
revised, each office should issue instructions to staff to ensure the recommendation is
timely implemented.
Ensuring Visual Aids Show Source of Data
We noted examples in OIG reports of tables, charts, and other visual aids that do not contain the
source of the information. As a result, the reader has difficulty assessing the source of
information provided in the visual aid. As a best practice when visual aids, such as tables,
charts, etc., are used in reports, the source should be named either in the text or in the credit line,
in small type, just below the illustrations. For example, if the table or chart is OIG-constructed,
it should be identified as such and an explanation provided as to where the data originated. The
12
-------
OIG Report Formatting and Style Guide and Report Quality Scoresheet for Draft Submissions do
not specifically address this issue.
Recommendation 11: OCPL should revise the Report Formatting and Style Guide and
Report Quality Scoresheet for Draft Submissions to require that the source of information
for all tables, charts, graphs, or other visual aids be clearly stated either in the report or in
the visual aid. Editors should check to ensure the source is provided during the editing
process.
OIG Actions Taken and Planned: OCPL will make changes to the Report Formatting
and Style Guide and to the Report Quality Scoresheet for Draft Submissions incorporating
this recommendation. Each office should also issue instructions to their staff to help
ensure this recommendation is timely implemented.
Describing the Approach for Each Objective in
Scope and Methodology
Audit results should be responsive to the audit objectives. Accordingly, the report should
describe in the Scope and Methodology section how each objective was addressed. We noted
that in one report it was very clear in the Scope and Methodology section as to how each
objective was developed. For each objective a paragraph began with the phrase "in order to
determine how OECA (objective 1 stated) we (then the report provides a description of the
comparison, analysis, or interviews made)." This step assists the reader in determining that
evidence obtained by the OIG was sufficient/competent and relevant to support the finding and
recommendations. In other reports the description of how each objective was addressed was not
clearly described.
Currently, the OIG'?, Report Formatting and Style Guide discusses methodology and states "the
methodology should address our general review approach, such as noting what types of
transactions we reviewed, as well as provide details on the analysis techniques we used (such as
statistical sampling)." The Guide should be revised to insert a statement that the review
approach be discussed by objective where feasible to assist the reader in judging the approach
and the whether the approach results in sufficient, competent, and relevant evidence to support
the finding.
13
-------
Recommendation 12: Amend the OIG's Report Formatting and Style Guide to instruct
staff to describe how each objective was addressed in the report's Scope and Methodology
section. Editors should also ensure the report clearly describes how each objective was
addressed. To avoid redundancy, reports should only list once those steps that address all
objectives.
OIG Actions Taken and Planned: OCPL will revise the Report Formatting and Style
Guide to state that the report should "describe the review approach by objective." To
avoid redundancy, the guide will direct writers to only list once steps that address all
objectives.
Each office should issue instructions to staff to help ensure timely implementation of the
recommendation until the OCPL Guide is revised.
14
-------
Chapter 4
Administrative Enhancements
Ensuring Staff Charge Time in a Uniform Manner
Over the past several years, one of the OIG's goals was to have professional staff charge 1,600
of the 2,087 work hours per year to direct time (specific assignments) or about 77 percent. The
remaining hours (about 500) were to be used for indirect time such as audit planning, training,
and sick leave.
In a March 1, 2006 email, OPE staff were directed to use newly created IGOR codes for
planning and other indirect charges as follows:
• Planning: "to be used for project work before initiation of preliminary research. Your
Assignment Manager and/or Product Line Director will notify you when this code is to be
used."
• Training: "to be used when you are in training unless that training is specific to a project
you are working on. So, for example, if you are doing the Data Mining module in IGEL,
you should use this code, but if you are attending a conference on small drinking water
systems for a job on small drinking water systems, you should charge your time in IGOR
against the project code."
• Management: "to be used for management activities such as creating an Individual
Development Plan, staff development, or other kinds of management activities."
• Supervision: "is only for the Product Line Directors and Assignment Managers. Use this
code for activities such as writing PERFORMS or giving performance feedback."
The former Acting AIG for OPE stated that, previously, the OIG had a lot of codes, some codes
were duplicative, and field staff had different codes or could create codes. She noted that there is
no OIG policy on IGOR codes and how staff should charge their time (direct or indirect). She
added that there is still a presumption that offices continue to develop their own approaches.
As a result, in March 2006, OPE managers established the four codes described above to capture
indirect time as well as direct time associated with assignments before a specific IGOR assignment
code is established. With respect to the planning code, the former Acting Director of OPE stated:
.. .the "planning" code captures direct time associated with assignment work because
it includes planning and research associated with assignments_where we haven't set
up an IGOR code. We don't set up "direct" IGOR codes for assignments until
notification memos go out. In my area, all staff time prior to getting that notification
memo sent out, that involves planning and research for the new assignment is charged to
planning. So not all planning charges are indirect.
15
-------
In one Resource Center, a review of 6 pay periods (Pay Periods 12-17) showed a fairly high
percentage of time charged by 6 of 15 OPE employees to the planning and management indirect
codes:
Table 4. Time Charged by Six OPE Employees to the Planning and Management Indirect Codes
Hours Charged
Percent of Total Time (480 hours)
Employee 1
99
20%
Employee 2
183
38%
Employee 3
86
18% (GS-14 Assignment Manager)
Employee 4
181
38%
Employee 5
217
45%
Employee 6
128
27%
In certain instances, staff may be performing general research and using the planning code is
completely appropriate. However, as noted above, direct planning time associated with specific
assignments that is charged to the planning code will not be charged to the IGOR code
established when the notification letter goes out. Therefore the actual cost of the assignment will
not be captured unless both codes are combined. Combining the two codes will also allow the
OIG to determine whether staff met the goal of 1,600 hours on assignments. Finally, the term
"other management activities" under management is ambiguous and needs to be better defined.
We found that OA did not have the exact same definitions for IGOR codes as OPE. We noted
that one OA employee in the same Resource Center for the same recent 6 pay periods charged 52
percent (250 hours) of indirect time to an IGOR code titled "administrative activities." The
auditor likewise explained that the time included preliminary research activities on two or three
assignments, online training, and other general research on potential audit issues.
Recommendation 13: OIG should issue a formal policy to (a) standardize the use and
definition of certain administrative/indirect time codes to better assess efficiency and the
true cost of operations; (b) formalize any time goals regarding time charging by staff; and
(c) ensure managers review time charges by staff for accuracy.
OIG Actions Taken and Planned: Various offices have worked on establishing a
uniform set of codes for capturing time or to ensure managers have access to time charges
of their staff.
The Acting Inspector General has asked the Office of Quality Assurance and Inspections
to develop a time policy in coordination with all OIG offices. That policy and any
associated goals for staff on charging time should be issued so that offices can implement
the policy beginning with FY 07. As noted in the tables, certain assignments can generate
more than one report. The policy will also address when teams should establish separate
job codes for each report where appropriate to enhance the OIG's ability of capturing
relevant costs associated with specific reports.
16
-------
Entering Performance Measurement and Results System
(PMRS) Data
During our review we noted that data were accurately entered - with four exceptions. For one
report, cost efficiencies of about $800 million were claimed and entered into PMRS. The final
report showed about $500 million in cost efficiencies. On two other assignments, involving
State Revolving Fund audits, the entries had not been made into PMRS. When contacted, the
audit teams for these three assignments made proper entries into PMRS. Finally, for one report,
PMRS shows the results for another report and the Director has been contacted.
For 11 of the 26 assignments, the QA Checklist was not completed for the Post Reporting
section, which asks whether results were entered into PMRS. Each Director should check and
ensure the QA Checklist is completed for Post Reporting. This step will help ensure accurate
entries are made in PMRS for all assignments and that the full impact of OIG work is captured.
Recommendation 14: Require each Director to review the QA Checklist at the end of
each assignment to ensure the QA checklist has been fully completed, including the
section for Post Reporting.
OIG Actions Taken and Planned: The next revision to the Project Management
Handbook will incorporate this recommendation. Until it is revised, each office should
issue instructions to their staff to ensure the timely implementation of this
recommendation.
17
-------
Appendix A
Project Quality Scorecard
The project quality scorecard objectively evaluates the activities of work that leads to the draft
reports that are submitted to OCPL for review. Once received by OCPL, the draft report is
scored using the OCPL Report Quality Scoresheet for Draft Submissions. Additional
information on that scoresheet is provided in Appendix B.
The following comments are provided to help the reader better understand how the elements in
the Project Quality Scorecard are measured:
Evidence
As stated in Section 7.50 of Government Auditing Standards, evidence may be categorized as
physical, documentary, testimonial, and analytical. The scoring system reflects the strength of
each type of evidence.
Physical evidence is obtained by auditors' direct inspection or observation of people, property, or
events. Such evidence may be documented in memoranda, photographs, drawings, charts, maps,
or physical samples.
Documentary evidence consists of created information such as letters, contracts, accounting
records, invoices, and management information on performance.
Testimonial evidence is obtained through inquiries, interviews, or questionnaires.
Analytical evidence includes computations, comparisons, separation of information into
components, and rational arguments.
18
-------
Project Quality Scorecard
Background Information
Report Title:
Report #
Date of Kickoff
Assignment #
Date of Entrance Conference
Total IGOR Days
Date of Draft Report sent to
OCPL for Review
Total Hours
Date of Draft Report
Project Cost
Date of Final Report
Significance Rating
Monetary benefits (each $1 million = 1 point)
Recommendations to change EPA policy or regulation (each 1 = 1 point)
Recommendations to implement new EPA policy or regulation (each 1 = 1 point)
Specifically answer a customer request (each 1 = 1 point)
Recommendation to Congress (each 1 = 1 point)
Evidence Rating
Evidence supporting the condition/main fact. Note: If there are multiple conditions
/main facts in an audit or evaluation, the score will be determined by averaging the
scores for each condition or main fact.
Documentary evidence (4 points)
Analytical (3 points)
Observation (3 points)
Testimonial (1 points)
Supervision Rating and Reviews
Note: The rating assigned to review comments/disposition and to the number of
supervision reviews are added for a net supervision score as follows:
Reviewer notes (compute score using the following steps)
a. Number of reviewer notes:
b. Number with comment, response, and acceptance by reviewer (b/a)
.00
c. Percentage x 100 divided by 4 = Supervision points for Reviewer Notes Score
Supervisory reviews (compute score using the following steps)
A. Identify number and grades of staff working on the audit/evaluation list:
Name Grade Level
1.
2.
3.
19
-------
4.
5.
B. Compute number of months of field work (date of kickoff to date of
message agreement meeting)
C. Number of Supervisory reviews that should occur:
( GS 5-9: 2 reviews a month) Number that occurred:
(GS 11-13: 1 review a month) Number that occurred:
D. Percentage of required reviews accomplished for all grade levels
E. Percentage of required reviews accomplished x 100/4
Points for Supervisory reviews
F. Sum of Reviewer notes and Supervisory review = Supervision rating points
N/A*
Report Phase
Number of days from kickoff to date draft report sent to OCPL for
review
Subtract: One point for each 50 days exceeding 200
Preliminary Research Guide
Preliminary research guide completed prior to kickoff meeting: Add 1 point
Fieldwork Guide
Fieldwork guide completed prior to entrance conference: Add 1 point
Finding Outlines
Finding outlines completed prior to Message Agreement Meeting: Add 1 point
Total Quality Score
Note: The score for supervision will be given a "weighted" score in future assignments when
supervision can be measured.
20
-------
Appendix B
Report Quality Scoresheet for Draft Submissions
The Acting Inspector General directed OCPL to develop a system to evaluate the quality of
incoming draft reports, providing OCPL with a Navy Audit report scoresheet, and directing
OCPL to include readability in the scoring. Given these parameters, the OCPL Publications Unit
created the Report Quality Score Sheet for Draft Submissions based on existing report
requirements and guidance included in the Project Management Handbook, the Report
Formatting and Style Guide, and writing principles taught in Write to the Point©. Once we
implement this scoring process, the Publications Unit will score incoming drafts during the
editing process.
While some of the elements of the Scoresheet can be objectively evaluated, objective criteria and
tools cannot address all the important elements of reports, such as organization, structure, clarity,
and the ability of the report to communicate the message. Therefore, the Publications Unit
included subjective measures in the Scoresheet to address whether the report elements are clear,
concise, convincing, logical, and relevant, and provide the proper perspective.
The Publications Unit assigned point values to the criteria so the total points would equal 100, to
be more easily incorporated into the overall scoring system. There is no direct correlation
between the number of requirements and the number of points possible, so the scoring is
subjective.
The Publications Unit assigned 30 points of the 100 points possible to meet the Acting Inspector
General's direction to emphasize the readability index. Readability indices are tools that help
determine how readable documents are. The Publications Unit chose the Flesch-Kincaid Index,
similar to the Fog Index, for three reasons: the index uses a fairly simple formula; the Federal
Government frequently uses the index; and it was developed based on adult training manuals, not
school textbooks. The formula considers the average number of words per sentence and the
average number of syllables per word.
The Publications Unit selects a portion of text from the At a Glance, the Introduction, and a
Finding Chapter. In cases where a report may not have all these sections, the Publications Unit
improvises. If the team writes consistently throughout a report, following the existing report
requirements, guidance, and templates, no concern over bias should occur.
While a good readability score does not ensure that a document is well written, it is an indicator
of the difficulty a reader will have understanding our message. Regardless of how complex an
assignment is, we need to explain our message in a manner that the uninformed reader will easily
understand. However, we took the technical nature of our reports into consideration by
proposing an educational grade level of 14 as our goal, which is equivalent to the New York
Times, as opposed to an educational grade level of 10, which is equivalent to Time or Newsweek
magazines. To improve readability and achieve a good score, writing teams need only write
shorter sentences, mix shorter sentences with longer ones, avoid words with several syllables,
and use plain language.
21
-------
Report Quality Scoresheet for Draft Submissions
Report Title:
Assignment Number:
Product Line Director:
Assignment/Project Manager:
Reviewer:
Review Date:
Total Score: 100
Preliminary Information
Requirements
Points
Possible
Points
Earned
Report Cover
- Is the cover in the proper format?
- Is the report title sufficiently descriptive yet concise?
- Is a position taken in the title?
- Is the assignment number included on the draft?
2
Inside Cover
- Are all abbreviations in the report included in the list?
- If there is a photo on the cover, is a caption included, with source?
1
At a Glance
- Is it in the proper format and confined to one page?
- Is the purpose of the report in the "Why...." section?
- Is the necessary perspective presented in the "Background" section?
- Is a "snapshot" of findings presented in the "What We Found" section?
- Are all the objectives addressed in the "What We Found" section?
- Are recommendations summarized in "What We Recommend" section?
5
Transmittal Memo
- Is it in the proper format?
- Is the template language used?
- Are phone and e-mail contacts listed?
1
Table of Contents
- Are the appropriate entries included, in the proper format?
1
Subtotal
10
Remarks:
22
-------
Introductory Information
(usually Chapter 1)
Requirements
Points
Possible
Points
Earned
Purpose
- Are the objectives clearly and concisely presented?
3
Background
- Is sufficient yet concise detail on what was reviewed provided?
- Are data provided for perspective (dates, dollars, quantities)?
- Are the responsible offices noted?
3
Scope and Methodology (including appendix information)
- Is the extent of the work performed to accomplish objectives noted?
- Are the universe and what was reviewed noted?
- Are the organizations visited and their locations noted?
- Is the period for when the review began and ended noted?
- Is the period of transactions covered noted?
- Are evidence gathering and analysis techniques described?
- Is review for compliance described, if appropriate?
- Is a sample design noted?
- Is the quality of data discussed?
- Is a Government Auditing Standards statement included?
4
Prior Coverage (can be part of "Scope and Methodology")
- Are the name, number, and date for prior audits provided?
- If no prior coverage occurred, is that acknowledged?
1
Internal Control (can be part of "Scope and Methodology")
- Is the scope of management control reviews noted?
- Are applicable management controls identified?
- Is what was found regarding internal controls noted?
- If internal controls were not reviewed, is that explained?
1
Subtotal
12
Remarks:
23
-------
Rest of Report
Requirements
Points
Possible
Points
Earned
Chapters/Findings
- Do chapter and section headings take a position and make sense?
- Is each finding organized clearly and logically?
- Are results and conclusions logical and concise?
8
"Charge" Paragraphs
- Is the charge paragraph for each chapter a reasonable length?
- Do they include condition, criteria, cause, and effect?
- Is the condition presented in the first sentence?
- Are all the objectives answered?
- Are the main points clear and concise?
8
Condition
- Is what was right, wrong, or needing improvement adequately discussed?
3
Criteria
- Are the criteria by which the condition was judged noted?
3
Cause
- Is the underlying reason for the condition identified?
3
Effect
- Is the ultimate effect on public health and the environment noted?
- Are quantities/potential cost benefits noted, when applicable?
3
Recommendations
- Are they action-oriented (avoiding weak words)?
- Do they address the underlying causes and weaknesses?
- Do they flow logically from the findings?
6
Status of Recommendations and Potential Monetary Benefits
- If there are any recommendations, is the table provided?
- Are all elements presented accurately?
2
Appendices
- Are they necessary?
- Are they clearly presented?
- Are they referenced in the report?
2
Subtotal
38
Remarks:
24
-------
Overall Formatting, Style, and Readability
Requirements
Points
Possible
Points
Earned
Is the Flesch-Kincaid Index lower than 14.0?
30
Does the report follow grammar rules and OIG writing guidance for elements such
as active voice, subject/verb agreement, capitalization, etc.?
5
Are the chapters and/or sections properly formatted?
3
Are tables/charts/photos properly numbered, labeled, and formatted?
2
Subtotal
40
Remarks:
Total Score
Sections
Points
Possible
Points
Allowed
Preliminary Information
10
Introductory Information
12
Rest of Report
38
Overall Formatting, Style, and Readability
40
Total
100
25
-------
Appendix C
Scope and Methodology
To perform our review we received printouts from the Office of Planning, Analysis, and Results
on OIG reports issued and also reports of time expended on the assignments. We then reviewed
the assignment workpapers in the OIG's AutoAudit® workpaper system and the final reports
using the Scoring Form attached as Appendix A. We also contacted supervisors as needed on
each assignment to obtain additional information. The Scoring Form measured each assignment
as to Significance, Evidence Rating, Supervision Rating and Reviews, Report Phase, Preliminary
Research, Fieldwork, and Finding Outlines. The OCPL Publication Unit developed a Report
Quality Scoresheet for Draft Submissions for assessing the quality of draft reports. We believe
these scorecards can be applied to all OIG assignments in accordance with the GAGAS standards
(be well written, timely, and have impact). The primary difference should be only in the type of
impact. The scorecards should allow for enough variety in impact quality measurement to cover
all of our work.
Accordingly, our scope covered final reports issued by OA, OPE, and OCPL from October 1,
2005, through March 31, 2006. We did not include single audit reports, DCAA contract audit
reports, or other reports where the work was performed by external auditors. This project did not
include the Audit of EPA 's Fiscal 2005 Financial Statements. We did not attempt to re-verify
the evidence supporting the report from an independent referencer perspective.
We did not score the "Significance" of the assignment unless the report clearly demonstrated that
the Agency had either fully implemented the recommendation or responded to a customer
request. Because of the manner in which workpaper review notes were maintained in the
workpapers, we were unable to score the Supervision Rating and frequency of supervisory
reviews.
Master List of OIG Products Reviewed for this Project
1. Rulemaking on Solvent Contaminated Industrial Wipes (OPE), Report No.
2006-P-00001, October 4, 2005.
2. EPA Could Improve Its Information Security by Strengthening Verification and
Validation Processes (OA), Report No. 2006-P-00002, October 17, 2005.
3. Changes Needed to Improve Public Confidence in EPA '.s Implementation of the Food
Quality Protection Act (OPE), Report No. 2006-P-00003, October 19, 2005.
4. 2006-P-00004-Ecology and Environment, Inc., Needs to Improve Information
Technology General Controls (OA), Report No. 2006-P-00004, November 22, 2005.
Report No. 2006-P-00005 - We did not score this report because the work was performed by
outside auditors (KPMG).
26
-------
5.
6.
7.
8.
9.
10
11
12
13
14
15
16
17,
18
EiM Performance Measures Do Not Effectively Track Compliance Outcomes (OPE),
Report No. 2006-P-00006, December 15, 2005.
More Information Is Needed On Toxaphene Degradation Products (OCPL), Report
No. 2006-P-00007, December 16, 2005.
Review of Complaint on the University of Nevada, Reno, Regional Environmental
Monitoring and Assessment Program Cooperative Agreement CR 826293-01 (OCPL),
Report No. 2006-P-00008, December 28, 2005.
Opportunities to Improve Data Quality and Children '.s Health through the Food
Quality Protection Act (OPE), Report No. 2006-P-00009, January 10, 2006.
Information Security Series: Security Practices - Integrated Contract Management
System (OA), Report No. 2006-P-00010, January 31, 2006.
EPA '.s and Mississippi's Efforts to Assess and Restore Public Drinking Water Supplies
After Hurricane Katrina (OPE), Report No. 2006-P-00011, February 14, 2006.
Office of Underground Storage Tanks Has Improved Contract Administration, But
Further Action is Needed (OA), Report No. 2006-P-00013, February 28, 2006.
EPA Can Better Manage Superfund Resources (OPE), Report No. 2006-P-00013,
February 28, 2006.
EPA '.s and Louisiana '.s Efforts to Assess and Restore Public Drinking Water Systems
after Hurricane Katrina (OPE), Report No. 2006-P-00014, March 7, 2006.
EPA Office of Air and Radiation and Office of Water Can Further Limit Use of Level
of Effort Contracts (OA), Report No. 2006-P-00015, March 14, 2006.
EPA Can Better Implement Its Strategy for Managing Contaminated Sediments
(OPE), Report No. 2006-P-00016, March 15, 2006.
EPA Can Improve Emissions Factors Development and Management (OPE), Report
No. 2006-P-00017, March 22, 2006.
EPA Provided Quality and Timely Information Regarding Wastewater after Hurricane
Katrina (OPE), Report 2006-P-00018, March 28, 2006.
Information Security Series: Security Practices - Comprehensive Environmental
Response, Compensation, and Liability Information System (OA), Report No.
2006-P-00019, March 28, 2006.
27
-------
19. Information Security Series: Security Practices - Integrated Compliance Information
System (OA), Report No. 2006-P-00020, March 29, 2006.
20. Information Security Series: Security Practices - Safe Drinking Water Information
System (OA), Report No. 2006-P-00021, March 30, 2006.
21. Federal Information Security Management Act-FY 2005 Status of EPA '.s Computer
Security Program (OA), Report No. 2006-S-00001, October 3, 2005.
22. Evaluation of the Effectiveness of EPA '.s Emergency Response Activities (OPE),
Report No. 2006-M-000004, February 24, 2006.
23. State of Nevada Drinking Water State Revolving Fund Program Financial Statements
for the Year Ended 6/30/2004 (OA), Report No. 2006-1-00018, November 29, 2005.
24. State of Oregon Clean Water State Revolving Fund Program Financial Statements for
the Year Ended June 30, 2005 (OA), Report No. 2006-1-00021, January 12, 2006.
25. State of Nevada Clean Water State Revolving Fund Program Financial Statements for
the Year Ended 6/30/2004 (OA), Report No. 2006-1-00024, January 23, 2006.
26. State of Illinois' Credit Claim for the Ottawa Radiation Site, Ottawa, Illinois (OA),
Report No. 2006-4-00026, October 31, 2005.
27. Mixed Funding Claim, Whitehouse Oil Pits Superfund Site, Duval County, Florida
(OA), Report No. 2006-4-00027, October 31, 2005.
28
------- |