U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Quality Assurance Report:
Assessing the Quality of the
Independent Referencing
Process During
Fiscal Year 2011
Report No. 12-N-0416
April 19, 2012

-------
Report Contributor:
Wanda M. Arrington
Abbreviations
AIG
Assistant Inspector General
DIG
Deputy Inspector General
EPA
U.S. Environmental Protection Agency
FY
Fiscal year
GAGAS
Generally accepted government auditing standards
OA
Office of Audit
OIG
Office of Inspector General
OMS
Office of Mission Systems
OPE
Office of Program Evaluation
PLD
Product Line Director
PM
Project Manager
QA
Quality Assurance
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
e-mail:	OIG Hotiirie@epa.aov	write: EPA Inspector General Hotline
phone:	1-888-546-8740	1200 Pennsylvania Avenue NW
fax:	202-566-2599	Mailcode 2431T
online:	http://www.epa.aov/oia/hotline.htm	Washington, DC 20460

-------
^tDsx
£ A \
V PRO^SO
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
12-N-0416
April 19, 2012
Why We Did This Review
To assess the quality of the
Office of Inspector General
(OIG) independent referencing
process, we surveyed Project
Managers (PMs) and their
respective Product Line Directors
(PLDs). The survey addressed
consistency among the Quality
Assurance (QA) staff, timeliness
of the reviews, best practices, and
areas of improvement.
Background
OIG Policy 006 states that the
quality of all worked performed
by the OIG significantly impacts
our credibility and effectiveness
in performing oversight functions
of U.S. Environmental Protection
Agency programs and activities.
Report indexes are reviewed (i.e.,
referenced) to provide reasonable
assurance that OIG reports are
factually supported in terms of
sufficiency, competency, and
relevancy of evidential matter.
Referencing provides reasonable
assurance that the reports comply
with generally accepted
government auditing standards.
Quality Assurance Report: Assessing the
Quality of the Independent Referencing
Process During Fiscal Year 2011
What We Found
Overall, the majority of PLDs/PMs who responded to our survey believe that
the independent referencing process is effective and efficient, and works well.
The respondents believe the referencers are consistent in the independent
referencing process. In addition, they believe the referencing process was
timely once the review began. Nonetheless, recommendations and
opportunities for improvements were identified by the PLDs/PMs.
What We Recommend
We made several recommendations to improve consistency and timeliness in
the referencing process. The QA staff agreed with most of the
recommendations. However, the QA staff did not agree with recommendations
where they believed independence would be compromised.
For further information, contact
our Office of Congressional and
Public Affairs at (202) 566-2391.
The full report is at:
www.epa.aov/oia/reports/2012/
20120419-12-N-0416.pdf

-------
^tDsrx
£	*'o
< H0? s	UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
\ V\l// ®	WASHINGTON, D.C. 20460
PRO^
OFFICE OF
INSPECTOR GENERAL
April 19,2012
MEMORANDUM
SUBJECT: Quality Assurance Report:
Assessing the Quality of the Independent Referencing Process
During Fiscal Year 2011
Report No. 12-N-0416 (
FROM: Wanda M. Arrington, Project(Manager/Lead Auditor
Quality Assurance Staff (on Detail)
Immediate Office
TO:
Arthur A. Elkins, Jr.
Inspector General
Attached is the report on assessing the quality of the Office of Inspector General's independent
referencing process for reports issued during fiscal year 2011. This report makes observations
and recommendations to you that will continue to enhance and strengthen the referencing
process.
If you have any questions regarding this report and planned corrective actions, please contact me
at (202) 566-2533.

-------
Quality Assurance Report: Assessing the Quality of the	12-N-0416
Independent Referencing Process During Fiscal Year 2011
	Table of Contents	
Purpose		1
Background		1
Scope and Methodology		1
Results of Review 		2
Consistency of Reviews		2
Timeliness of Reviews		4
Best Practices		5
Areas of Improvement		6
Recommendations		6
QA Team's Response to Recommendations		7
Other Considerations		8
Summary of QA Team's Response to Other Considerations		9
Appendix
A PLD/PM Survey Responses	 10

-------
Purpose
To assess the quality of the U.S. Environmental Protection Agency (EPA) Office
of Inspector General (OIG) independent referencing process, we conducted a
survey of Project Managers (PMs) and their respective Product Line Directors
(PLDs). The survey addressed consistency among the Quality Assurance (QA)
staff (i.e., referencers) and timeliness of the reviews. Suggestions for areas of
improvement were also identified.
Background
As stated in OIG Policy 006, the quality of all worked performed by the OIG
significantly impacts our credibility and effectiveness in performing oversight
functions of EPA programs and activities. One operational activity performed to
help fulfill and verify the requirements of quality is referencing. Referencing
entails reviewing the workpaper support identified through indexes for
correctness to ensure the quality and accuracy of OIG reports. Report indexes are
reviewed to provide reasonable assurance that OIG reports are factually supported
in terms of sufficiency, competency, and relevancy of evidential matter.
Referencing of OIG audit and evaluation products provides reasonable assurance
that they comply with generally accepted government auditing standards
(GAGAS).
OIG Procedure 006 outlines the referencing standard operating procedures.
According to OIG Procedure 006, PLDs/PMs should expect the referencing
review (from start through the clearance of comments) to take approximately
10 business days from when the referencer notifies the PLD/PM that the review is
about to commence. The referencing review of a final report will usually take
approximately 2 business days (assuming there are only a minor number of
changes between the draft and final). However, depending upon the length of the
Agency response and the corresponding OIG analysis, the referencing review of a
final report could take up to 5 business days.
Scope and Methodology
We conducted a survey with selected PMs and their respective PLDs to maintain
consistent questioning and identify positive and negative practices within the
referencing process. Our scope covered reports issued by the Office of Audit
(OA), Office of Program Evaluation (OPE), and Office of Mission Systems
(OMS) during fiscal year (FY) 2011. The scope of work performed does not
constitute an audit under GAGAS.
The PMs were selected for this survey if they had at least one report issued in
FY 2011 that was independently referenced by each member of the QA staff
(i.e., three or more reports). A total of six PMs and the five corresponding PLDs
from OA and OPE participated in this survey. We did not identify any PMs from
12-N-0416
1

-------
OMS meeting the criteria. The survey covered 21 of the 81 (26 percent) reports
reviewed by the QA staff during FY 2011. For a breakdown of survey selection,
see appendix A.
Results of Review
Overall, the majority of PLDs/PMs believe the independent referencing process is
effective and efficient, and works well. The respondents believe the referencers
were consistent in the independent referencing process. In addition, they believe
the referencing process was timely once the review began. Nonetheless,
recommendations and areas of improvement were identified regarding
consistency and timeliness.
Consistency of Reviews
Seventy-three percent (8 out of 11) of the respondents believed the referencers
were consistent with the types of comments made. Respondents also believe that
the referencing comments were readily understood and clear, and that they could
always talk with the referencers to obtain clarification. Overall, respondents
believe the referencers consistently responded in a professional and respectful
manner, and complied with OIG policies and procedures, as shown in figure 1.
Figure 1: PLD/PM survey responses—consistency of reviews
Consistency of IR comments made
Comments readily understood/clear
Amount of report referenced appropriate
Responded in professional & respectful manner
Handling of unresolved issues
Complied with the OIG policy & procedures
Applied OIG Policy and Procedures 006
Confident report facts adequately supported
Comments added/improved the report quality
IR helped team comply with GAGAS
# OF RESPONDENTS 0 2 4 6 8 10 12
¦ YES ¦ NO BOTHER
Source: OIG analysis of PLD/PM survey responses.
The following summarizes the PLD/PM feedback on the consistency in the
independent referencing process:
• Even though a majority of the PLDs/PMs believe there was consistency in
the types of referencing comments made, there was an overwhelming
consensus that there are some differences in the style or approach among
12-N-0416
2

-------
the referencers and how they handle unresolved issues. However, many
acknowledged that differences are to be expected based on individuality or
personality (e.g., some were more thorough and tougher to clear
comments; some were more flexible).
•	There was a difference in the type of indexing the referencers expected to
see in workpaper support (e.g., indexing a spreadsheet). Referencing
comments appeared to be based on preference rather than unsupported or
incorrect information in the report, and a subjective interpretation of
sufficient and appropriate evidence for findings and recommendations in
the report.
•	Referencing comments went beyond the scope of determining whether the
report was adequately supported by the indexed workpapers. Referencing
comments were editorial and questioned the scope of work, execution of
steps, or approach taken.
•	Instead of documenting every referencing comment, some referencers call
the PLD/PM to discuss an issue beforehand to obtain clarity, which they
liked.
•	Some respondents do not believe an appropriate amount of the report was
referenced. One respondent said the referencers spend too much time
referencing and review too much, which the respondent believes is outside
the scope of referencing. The teams have a tendency to provide more
detailed indexing with the hope of reducing the number of referencing
comments, which does not appear to help. Another respondent said the
referencers spent a disproportionate amount of time on a short letter report
compared to other reports. Another responded that although it is not
transparent what the referencer looked at, nothing of significance was
missed during the independent referencing process.
•	Even though respondents believe the referencers complied with OIG
policies and procedures, a couple of respondents thought the policies and
procedures were general, not clear and detailed, and inconsistent with the
role of independent referencing. OIG referencing is much broader and
includes more than just checking workpaper support. It includes checking
sampling criteria and methodology, workpaper supervisory review, and
workpaper content (e.g., purpose, source, scope, conclusion), which
respondents said are the responsibility of the team, PM, PLD, and
Assistant Inspector General (AIG)—not the referencers.
•	After going through the independent referencing process, some
respondents felt confident that the facts in their reports were adequately
supported and the quality of their reports was improved. One respondent
12-N-0416
3

-------
said that this control process is valuable and needed. Another said it is
good to get someone who has not been involved in the assignment to
independently look at the evidence with a critical and skeptical eye. The
respondent likes the idea of an independent group referencing reports
compared to how we used to obtain staff for referencing. However, other
respondents questioned the cost benefit of the process, and did not believe
the time and money spent on some reports was worth the added value or
that the process was necessary.
• There were varying opinions about whether independent referencing
helped the team comply with GAGAS. Approximately half of the
respondents said that the team complied with GAGAS prior to the
independent referencing process. They described the independent
referencing process as an internal control to provide assurance that the
report is supported and in compliance. One respondent said that the
current process adds tremendously more value in content and substance
compared to the previous referencing process when a GS-13 was
referencing a report.
Timeliness of Reviews
Ten out of 11 respondents believed that the referencing was completed within a
reasonable amount of time given the size and complexity of the report, as
illustrated in figure 2. However, some respondents expressed that after the report
was submitted to the QA staff for referencing, the wait time was too long
(i.e., "in the queue") before a referencer started reviewing their report. Some
respondents were sympathetic and acknowledged the QA staff workload. One
respondent believed that bringing in staff to assist the QA staff with their
workload will help resolve this delay.
Figure 2: PLD/PM survey responses—timeliness of reviews
12
10
8
6
4
2
0
Completed in
reasonable time
Longer than the
stated goal
Extenuating factors
contributed to
untimeliness
NO
YES
Source: OIG analysis of PLD/PM survey responses.
12-N-0416
4

-------
In addition, based on the survey:
•	5 out of 10 respondents said the time for referencing the report was
longer than the goal stated in OIG Policy and Procedure 006, which is
10 business days for the draft report and 2 days for the final report.
•	4 out of 10 acknowledged that there were extenuating factors that
contributed to the referencing not being completed within the stated goal.
As previously mentioned, the referencers' large workload was a
contributor. In addition, the length and complexity of the reports and
resolution of comments were factors.
Best Practices
Seventy percent (7 out of 10 respondents) believe that the independent
referencing process is effective, efficient, and works well. Some things that work
well according to these PLDs/PMs include:
•	Having the opportunity to discuss issues with the referencer before the
comments were documented.
•	Communication, accessibility (before and during the independent
referencing process), and resolution (e.g., providing options or
suggestions). The referencers provided feedback to the managers so they
understood the nature and types of issues. They were willing to discuss
comments and explain their positions. In addition, the referencers, in some
cases, provided suggestions to address the comments or what they
believed could be done to remedy the referencer comment.
•	Appropriate priority being placed on getting the referencing done quickly,
and the referencer working well with the team to resolve any issues.
•	The objectivity of the referencers and clarity of their comments.
Thirty percent (3 out of 10 respondents) believed that the process was not
effective and/or efficient. These respondents' verbatim comments were:
•	"It is effective because it is added assurance that what is in the report is
supported. But it is not efficient because everything is duplicative going
through at least four or five levels of review. You have the team indexes,
PM, PLD, & QA checking indexes, there's editing, DIG/AIGA review,
legal, etc."
•	"The time required for the teams to get ready for independent referencing
and the time spent independent referencing indicate inefficiency."
12-N-0416
5

-------
• "The current process slows down issuing reports because the majority of
the comments appear to be based on subjective preference rather than
substantive issues with lack of support. Addressing these comments delays
issuing the report. In addition, teams spend additional time indexing the
report prior to submitting the report to QA by attempting to over index and
avoid unnecessary independent referencing comments."
Appendix 1 provides a complete breakdown of PLD/PM survey results.
Areas of Improvement
The respondents provided many recommendations for improving the independent
referencing process. When suggestions for improvements were identified by more
than one PLD/PM, they were included as recommendations addressing
consistency (recommendations 1 through 5) and timeliness (recommendations
6-8). Improvements to the independent referencing process identified by only one
PLD or PM did not warrant elevation to a recommendation, but are included as
other considerations for improvement.
Recommendations
1.	Provide guidance to OIG staff on indexing workpapers to support the
audit/evaluation reports.
2.	Limit the scope of the referencing review to determine solely whether the
indexing reasonably supports the audit/evaluation report.
3.	Communicate with the PLD/PM, as needed, during the referencing
process for clarification before providing referencing comments.
4.	Provide suggestions or alternatives for addressing referencing comments.
5.	Revise the descriptions of Agency agreement with findings and actions on
Agency responses.
6.	Get involved earlier in the assignments to improve the quality during
preliminary research and field work; identify earlier who the assigned
referencer will be; and meet with the team before referencing starts to
understand the subject matter of the report.
7.	Reduce the "wait time" before referencing begins and start reviewing the
report shortly after it is submitted for independent referencing.
8.	Encourage teams to submit the indexed audit/evaluation report in
manageable sections for referencing, and provide referencing comments
daily to the team to improve efficiency and productivity.
12-N-0416
6

-------
QA Team's Response to Recommendations
We appreciate all feedback received from the PLDs/PMs for this survey and plan
to address the areas for improvements identified. Referencing is a valuable
control, and we want to continue to improve the efficiency of this process.
1.	We agree to provide indexing tips and post to the Auditor's Corner on the
OIG Intranet site http://oigintra.epa. gov/auditors corner.cfm within
45 days from report issuance. Indexing tips are also included in the Project
Management Handbook.
2.	We appreciate the comments on the scope of the review. OIG Policy and
Procedure 006 already address this recommendation. Both the Policy and
Procedure 006 were reviewed by the AIGs and approved by the Inspector
General. In accordance with OIG Policy and Procedure 006, the QAteam
performs selective referencing of the audit/evaluation report. However, we
are mandated by the Government Auditing Standards and OIG policy to
perform quality assurance. Part of this function is accomplished through
the referencing process. According to the standards, referencing is a
quality control process to ensure the accuracy of reports. In addition, the
standards state, "Referencing is a process in which an experienced auditor
who is independent of the audit checks that statements of facts, figures,
and dates are correctly reported, that the findings are adequately supported
by the evidence in the audit documentation, and that the conclusions and
recommendations flow logically from the evidence." As outlined in Policy
and Procedure 006, referencing encompasses more than just verification of
facts. The QA staff will continue to include comments that improve the
quality of the report.
3.	We will continue to contact the PLD/PM on an as needed basis to obtain
clarification. In addition, we continue to encourage the PLD/PM to contact
the referencers if there are special considerations that should be brought to
their attention.
4.	We agree that providing suggestions can speed up the process; however,
we need to assure that QA staff maintains their independence. When
possible, the QA staff will attempt to offer suggestions for quick
resolution, such as providing alternative supporting workpapers or
language consistent with the workpaper support.
5.	We agree that the descriptions of Agency agreement with findings and
actions taken need revision and clarification. We are currently addressing
this issue as part of the 2750 workgroup, and the decision will be shared
with the OIG when the issue is resolved.
12-N-0416
7

-------
6.	We do not believe it is appropriate, from an independence perspective, to
be involved at the onset of an assignment. However, we are always
available and willing to answer any questions from OIG staff and provide
suggestions for improving the indexing to workpapers supporting the
report. Referencers are assigned to review reports based on when they
become available; therefore, it is not feasible or prudent to assign a
referencer earlier. We encourage OIG staff to inform us in advance of
special circumstances such as date mandated reports to better facilitate the
referencing review. In addition, occasionally the QA team may contact the
team at the start of referencing to identify any unique qualities about the
assignment and report.
7.	We agree there is an opportunity for efficiency by reducing the wait time
before we start referencing the report. We appreciate and take very
seriously your concerns in this area. We are conscious of the wait time and
will continue to make every effort to get the reports done as quickly as
possible, given all other priorities.
8.	We continue to encourage teams to submit complete, indexed reports in
manageable sections to allow concurrent referencing review of one section
and addressing referencer comments in another, as stated in OIG
Procedure 006. Where we can, we also notify the team that we are out of
the workpapers for the day so that the team's responses can be added to
the workpapers. Some teams also prepare responses that they can just cut
and paste in the workpapers on a daily basis when the referencer is
finished or has completed his or her review of the workpaper. Breaking up
the reports into sections may also assist in teams being able to respond
more timely to referencer comments.
Other Considerations
1.	Have at least one QA team member working, when feasible, to ensure
sufficient coverage for referencing reviews.
2.	Streamline the referencing process and establish different levels of
referencing reviews based on report type or subject matter significance.
3.	Eliminate independent referencing of the final report if there are no
significant changes, and allow minor changes to be reviewed at the PLD
level.
4.	Obtain more staff to assist with the QA team's workload.
12-N-0416
8

-------
Summary ofQA Team's Response to Other Considerations
In response to consideration 1, we make every effort to have at least one QA
team member working. When fellow team members are either on leave and/or
in training, another referencer will complete assigned work as long as we are
informed. Reports or questions should be addressed to the OIG Referencers
mailbox, which includes all referencers. Rare or extenuating circumstances
affecting availability exist, such as inclement weather, all QA staff may be in
training, leave during the holidays, or illness.
In response to considerations 2 and 3, we believe the signing of OIG Policy
and Procedure 006 has provided the OIG with structure for referencing
reviews. Streamlining the referencing process by establishing different levels
of review and not reviewing the final report, even if there are no substantial
changes, are not feasible alternatives. The referencers are responsible for
assuring GAGAS compliance, to include the accuracy and quality of our
written products regardless of the subject matter, size or type of report, or
intended audience. Generally, all reports are posted on the OIG website for
public viewing; therefore, it is imperative that we assure factual accuracy and
adequate support of all reports.
In response to consideration 4, for the immediate future, we have obtained
three detailed OIG staff to assist with the QA workload. We recently obtained
an additional three detailed OIG staff to assist with the external audit peer
review we will be conducting this year. However, we do not have the
authority to hire additional staff; therefore, we cannot provide a response on
acquiring additional and/or permanent staff for the QA team. We encourage
staff to participate in all detail opportunities with the QA team.
We consider our actions as satisfying the intent of the recommendations and
other considerations.
12-N-0416
9

-------
Appendix A
PLD/PM Survey Responses

PLD/PMs Responses
Questions
Yes
No
Other Response
Do you believe the QA staff was consistent with
the types of IR comments made on your report
(i.e., did they provide similar comments during the
IR process)?
8
73%
3
27%


Do you believe the IR comments were readily
understood or clear?
9
82%
1
9%
Yes & No -1
9%
Do you believe an appropriate amount of the
report was referenced?
8
73%
2
18%
Yes & No -1
9%
Do you believe the QA staff was consistent in
responding in a professional and respectful
manner?
11
100%
0
0%


Do you believe the QA staff was consistent in
handling unresolved issues?
7
64%
3
27%
Don't Recall -1
9%
Do you believe the QA staff complied with the OIG
policy and procedures?
10
91%
0
0%
Probably -1
9%
Do you believe they were consistent in applying
OIG Policy and Procedures 006 during the IR
review process? 1
10
100%
0
0%


After going through the IR process, do you feel
more confident that the facts in your report(s)
were adequately supported?
9
82%
2
18%


Do you believe the IR comments added to or
improved the quality of your report(s)?
8
73%
3
27%


Does the independent referencing help the team
comply with GAGAS?
7
64%
4
36%


Do you believe the independent referencing was
completed within a reasonable amount of time
(given the size and complexity of your report)?
10
91%
1
9%


Was the time for referencing your report longer
than the goal stated in OIG Policy & Procedures
006? 1
5
50%
5
50%


Were there any extenuating factors that
contributed to the referencing not being completed
within the aforementioned time? 1
4
40%
6
60%


Do you believe the IR process is effective and
efficient? 1
7
70%
3
30%


Do you believe the IR process worked well? '
7
70%
3
30%


Are there any areas during the IR process where
you believe the QA staff could be more consistent
in their review? 1
5
50%
5
50%


Is there anything that could improve in the process
to lessen the burden on your team? 1
9
90%
1
10%


1PLD did not answer the question and/or deferred to PMs who were better equipped to respond since the PLD had delegated
most of the responsibility to the PMs.
Total Reports (21)
PM Breakdown (6)
15-OA
<
o
"3"
6-OPE
2-OPE
0-OMS
0-OMS
12-N-0416
10

-------