OFFICE OF RESEARCH AND DEVELOPMENT
REPORT ON THE AIR POLLUTION PREVENTION
CONTROL DIVISION
NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
RESEARCH TRIANGLE PARK, NORTH CAROLINA
ORD MANAGEMENT REVIEW
OCTOBER 1996
-------
This report contains the findings of the Management Review of the Air Pollution
Prevention and Control Division, National Risk Management Research Laboratory, in
Research Triangle Park, North Carolina. The review was conducted October 20 -25,
1996, by the EPA staff listed below.
We certify that the contents of this report are accurate and based on information
obtained from review of documents provided, interviews of APPCD staff and file
reviews.
MANAGEMENT REVIEW TEAM LEADERS:
^
a I
i
NRMRL, AOS
ler
Walt Stutts, Extramural Management
Review Team Leader, ORMA, NERL
-------
TABLE OF CONTENTS
EXECUTIVE SUMMARY
BACKGROUND 1
GENERAL COMMENTS 1
PRINCIPAL FINDINGS ; i
CONCLUSION 2
RESPONSE TO DRAFT REPORT 2
CHAPTER 1 - INTRODUCTION
1.1 PURPOSE 4
1.2 BACKGROUND 4
1.3 SCOPE AND METHODOLOGY 5
CHAPTER 2 - GENERAL MANAGEMENT
2.1 BACKGROUND ; 6
2.2 STAFFING ISSUES 6
2.2.1 MORALE 6
2.3 FUNDING ISSUES 6
2.4 MANAGEMENT ACCOUNTABILITY 7
2.5 ETHICS .; 8
2.6 DIVISION MANAGEMENT ISSUES 8
CHAPTER 3 - FINANCIAL MANAGEMENT
3.1 INTRODUCTION GENERAL 10
3.2 FUNDS CONTROL
3.2.1 INTRODUCTION 10
3.2.2 REVIEW OBSERVATIONS AND FINDINGS 10
3.2.3 RECOMMENDATIONS 11
3.3 TRAVEL
3.3.1 INTRODUCTION 12
3.3.2 REVIEW OBSERVATIONS AND FINDINGS
3.3.2.1 LOCAL TRAVEL 13
3.3.2.2 DOMESTIC TRAVEL 13
3.3.2.3 REIMBURSABLE TRAVEL 14
3.3.2.4 INTERNATIONAL/REIMBURSABLE TRAVEL . 14
3.3.2.5 INTERNATIONALTRAVEL 14
3.3.3 RECOMMENDATIONS.
3.4 TIMEKEEPING
3.4.1 INTRODUCTION 15
3.4.2 REVIEW OBSERVATIONS AND FINDINGS 15
-------
3.4.2.1 PROCESS 15
3.4.2.2 RECORDKEEPING 16
3.4.2.3 TRAINING .'.' 16
3.4.2.4 SUPERVISORY APPROVALS 16
3.4.2.5 DESIGNATED AGENT (DA) 17
3.4.3 RECOMMENDATIONS 17
3.5 BANKCARD
3.5.1 INTRODUCTION 17
3.5.2 REVIEW OBSERVATIONS AND FINDINGS - BANKCARD
HOLDERS , 18
3.5.2.1 FILE MAINTENANCE AND DELEGATION OF
AUTHORITY , 18
3.5.2.2 PROCUREMENT REQUEST ., .'•'. 18
3.5.2.3 RECONCILED STATEMENT OF ACCOUNT .... 19
3.5.2.4 SUPPORTING RECEIPTS ....:..'. 20
3.5.2.5 BANKCARD/LOG 21
3.5.2.6 SUGGESTED IMPROVEMENTS BY THE
BANKCARD HOLDER 21
3.5.3 RECOMMENDATIONS 22
3.5.4 REVIEW OBSERVATIONS AND FINDINGS - BANKCARD
APPROVING OFFICIALS 23
3.5.4.1 DESIGNATION OF APPROVING OFFICIALS ... 23
3.5.4.2 TRAINING 23
3.5.4,3 PURCHASE REQUESTS, AUTHORIZATION,
RECORDING 23
3.5.4.4 RECONCILIATION OF BANKCARD STATEMENT 24
3.5.4.5 PERFORMANCE STANDARDS 24
3.5.4.6 SUGGESTIONS FOR IMPROVEMENT BY
APPROVING OFFICIALS 24
3.5.5 RECOMMENDATIONS 24
CHAPTER 4 - EXTRAMURAL MANAGEMENT
4.1 INTRODUCTION - GENERAL 26
4.2 CROSS-CUTTING ISSUES
INTRODUCTION... 27
1. FILES DO NOT CONSISTENTLY REFLECT APPROPRIATE
REVIEW AND APPROVALS 27
FINDINGS 27
A. EXAMPLES - "UNDOCUMENTED" AREA 27
B. EXAMPLES - "OVERLOOKED AREA" 28
2. DECISION MEMOS LACK KEY ELEMENTS 29
FINDINGS 29
A. PRINCIPAL PURPOSE FOR COOPERATIVE
AGREEMENTS 30
-------
B. SUBSTANTIAL INVOLVEMENT FOR COOPERATIVE
AGREEMENTS 30
C. PEER REVIEWS AND RECONCILIATION OF
CONTRARY REVIEWS '.'.' 31
D. APPARENT OR DUPLICATIVE EFFORT 31
E. STATUTORY AUTHORITY AND ORD ISSUE 31
F. QUALITY ASSURANCE REQUIREMENTS 32
G. COMPETITION 32
H. TRAVEL 32
3. RECORDS MANAGEMENT
FINDINGS 33
A. GOOD RECORDS MANAGEMENT PRACTICES .... 33
B. RECORDS MANAGEMENT PRACTICES THAT
COULD BE IMPROVED ;...... 33
4. DIVISION PERSONNEL DEMONSTRATE EFFECTIVE ," •
MONITORING/ADDRESSING PROBLEMS 35
5. LACK OF FINANCIAL DETAIL INHIBITS FINANCIAL
MONITORING OF CA'S AND lAG'S 36
FINDINGS 36
RECOMMENDATIONS 37
4.3 COOPERATIVE AGREEMENTS
INTRODUCTION 40
FINDINGS 40
1. SCOPES OF CA'S MODIFIED WITHOUT ORD AND/OR
GAD APPROVAL , 40
2. ASSISTANCE AGREEMENTS IMPROPERLY USED TO
ACQUIRE DIRECT AGENCY SUPPORT 42
3. DOCUMENTATION OF COMPETITION UNDER CA'S
COULD BE IMPROVED , 44
4. WEAK PEER REVIEWS MINIMIZE VALUE OF INPUT 47
5. ROOT CAUSE: FUNDING NON-SPECIFIC RESEARCH
PROJECTS 49
RECOMMENDATIONS 50
4.4 INTERAGENCY AGREEMENTS
INTRODUCTION 53
FINDINGS 53
1. SOME STATEMENTS OF WORK FOR lAG'S WERE
WELL DEVELOPED--QTHERS NOT 53
2. FRONT-END BUDGET DEVELOPMENT LACKING
FOR NEW lAG'S 55
RECOMMENDATIONS 56
4.5 CONTRACTS
INTRODUCTION 56
FINDINGS 57
1. WAM SUPPORT TO PROJECT OFFICER UNDER
ACUREX COULD BE IMPROVED 57
-------
RECOMMENDATIONS 58
EXHIBITS
# 1 - ADDITIONAL DETAIL RELATED TO (JR822870
# 2 - APPCD's RESPONSE TO DRAFT ORD MANAGEMENT
REVIEW REPORT (W/ATTACHMENTS)
# 3 - MEMORANDUMS
(3-1) Financial Management Team Evaluation of APPCD's
Response
(3-2) Extramural Management Team Evaluation of
APPCD's Response
,'
APPENDICES
APPENDIX A
APPCD ORGANIZATIONAL CHART
APPENDIX B
MANAGEMENT REVIEW TEAM
APPENDIX C
APPCD DELEGATED AUTHORITY
-------
EXECUTIVE SUMMARY
BACKGROUND
The National Risk Management Research Laboratory (NRMRL) conducted a
management review of the Air Pollution Prevention Control Division (APPCD), in Research
Triangle Park, North Carolina the week of October 20-25, 1996. The review was conducted in
accordance with the ORD Management Review Program established September 1994, and
covered the fiscal resource management activities of the APPCD.
The object of this review was to assess whether APPCJD management and staff were
properly using available funds and extramural instruments in compliance with applicable laws,
regulations, and Agency and ORD policies and procedures. The review also included an
assessment of management controls and procedures in the Division to determine whether they
were functioning effectively.
In the entrance conference, the review team met with the Division Director and senior
staff to discuss general management issues relevant to APPCD. This discussion covered staffing
and personnel, funding, management integrity, financial management, and extramural
management issues. The review team interviewed Laboratory management staff, administrative
and support personnel, project officers, and work assignment managers. Extensive file reviews
were also conducted by the review team. An exit conference was held by the review team
members with the Division Director and senior staff. Discussions centered around areas
reviewed and concerns raised during the review.
GENERAL COMMENTS
(• ., . -
FINANCIAL MANAGEMENT
APPCD management and staff exhibit a strong recognition and understanding of the
importance of management controls in this area. The financial management review team did note
however, that more detailed records would eliminate valuable time if an outside audit were to
occur.
PRINCIPAL FINDINGS
EXTRAMURAL MANAGEMENT
CROSS-CUTTING ISSUES
The review team identified certain weaknesses associated with various types of
agreements that have been identified in other management reviewes; namely, some files did not
-------
contain evidence of the appropriate level of approval, some of the decision memoranda were not
fully developed, and some files were not complete. Conversely, many of the files did
demonstrate the appropriate level of approval, did contain well developed decision memoranda
and did contain required documentation, including some that we're exemplary relative to
monitoring activities. Where appropriate, recommendations were made for corrective actions.
COOPERATIVE AGREEMENTS
The review team identified a number of common weaknesses among the assistance
agreements that were reviewed, including instances where CAs were modified without ORD
and/or GAD approval and where CAs were apparently used to acquire direct support. In
addition, the review team believes that there is room for improvement in documenting the
competition for assistance awards and in securing better peer reviews. The review team
concluded that the root cause leading to these problems was the award of CAs based upon
broadly worded, non-specific, research narratives which were supplemented after award by the
submission of work plans developed collaboratively by EPA and recipient personnel. The
review team provided a number of recommendations to address these issues, including the
recommendation to evaluate the practice of calling for post-award wqrkplans in light of the
concerns identified in this report,
INTERAGENCY AGREEMENTS
The review team again found some problems common to other management reviews in
the area of lAGs; namely that some of the SOWs were under-developed and there was a lack of
front-end budgets for hew agreements. Some SOWs were well developed, and could be used as
an example for other agreements.
CONTRACTS
/
The review team identified few problems with the contracts reviewed, with the primary
weakness being a lack of support by WAMs of the PO under a major technical support contract.
The review team believes this problem can be readily addressed, if the Division has not already
taken appropriate action.
CONCLUSION
Efficient, more detailed record keeping would enable APPCD to better manage financial
and extramural management resources. It is recommended that a corrective action plan be
developed, implemented and overseen by NRMRL management to ensure proper use and
management of extramural vehicles.
APPCD RESPONSE TO THE DRAFT REPORT
The Division Director responded to the draft report by memorandum. In order to
-------
maintain the natural flow of the report, these comments have been inserted where appropriate
behind each subject matter. In addition, a copy of the memorandum from the Division Director
in its original state has also been included at the end of the report.
t i
Generally, the Division's response lacked a fully developed plan to document the process
and report the results to NRMRL management. There is little indication of who will do what and
when they will do it. As a result, accountability is vague except that the Division Director will
see that the effort is completed. Again, it is recommened that NRMRL management solicit a
more fully developed action plan to address the recommendations made by the review team.
The extramural review team has noted their exceptions in a memorandum dated July 2,
1997; Subject: Evaluation of APPCD's Response to the Section 4 of the Draft ORD Management
Review Report (March 1997).
The financial management review team has noted two exceptions which have been
outlined in a memorandum dated June 6, 1997; Subject: Evaluation of APPCD's Response to
Financial Management Review, Section 3 of the Draft ORD Management Review Report (March
1997).
-------
CHAPTER 1 - INTRODUCTION
1.1 PURPOSE
In accordance with the ORE) Management Review Program, a management review was
conducted of the Air Pollution Prevention Control Division (APPCD), National Risk
Management Research Laboratory (NRMRL), in Research Triangle Park, North Carolina. The
management review team consists of personnel from the Office of Research and Development
(ORD) and the Office of Adminstfation (OA). The data gathering protion of the review occurred
October 20-25, 1996.
The object of this review is to assess whether APPCD management and staff are properly
using available funds and extramural instruments in compliance with applicable laws,
regulations, and Agency and ORD policies and procedures. The reviews also include an
assessment of management controls and procedures in the Division to determine whether they
are functioning effectively.
1.2 BACKGROUND
The mission of the Air Pollution Prevention Control Division reads, "develops and
assesses methods and technologies for characterizing emissions, and preventing or reducing the
deleterious effects of air pollutants on human health and welfare, and on the global
environment".
APPCD is comprised of five branches (1) Emissions Characterization and Prevention, (2)
Air Pollution Technology, (3) Technical Services, (4)Atmospheric Protection, and (5) Indoor
Environment Management. The Division conducts fundamental and applied research to: (1)
Develop emission methodologies and models; (2) Characterize and evaluate sources and
technologies for preventing or controlling volatile organic compounds and hazardous air
pollutants; (3) characterize and verify the performance of alternative energy technologies; (4)
evaluate technologies and systems contribution to stratospheric ozone depletion; (5) characterize
and evaluate sources of indoor environmental pollution; (6) develop, apply, and verify the
performance of combustion modification techniques; and (7) develop and verify the performance
of innovative technologies.
APPCD's FY 1996 FTE allocation and utilization was 85. Two vacancies were recorded.
In addition, APPCD had 50 ori-site contractors, 4 off-site contractors and 22 Senior
Environmental Employment Program enrollees. The table in Exhibit 1.1 shows the breakdown of
on-site personnel. An organization chart and staffing chart of the federal employees are included
in Appendices A-1 and A-2.
APPCD managed funds in two appropriations during FY 1996. S&T which includes the
4
-------
following media: air, hazardous waste, and multimedia and, Reimbursements. Total funds were
$22,471,044, including $312,789 carryover dollars.
1.3 SCOPE AND METHODOLOGY
This management review covers FY 1996 fiscal resource management and internal
control activities of the Air Pollution Prevention Control Division. This includes the financial
management areas of funds control, travel, bankcard, and timekeeping; the extramural
management areas of contracts administration, cooperative agreements, and interagency
agreements, and the general management area which covered ethics, internal control and FMFIA.
The overall objective of the review is to assess whether APPCD management and staff
are properly using available funds and extramural vehicles in compliance with applicable
statutes, regulations, and Agency and ORD policies and procedures. The review also includes an
assessment of Division management controls and procedures to determine whether they are
functioning effectively.
To prepare for the review, the team was provided with background information; previous
audit and management reports and related corrective action plans, financial management reports,
contract statements of work, the Division's year-end FMFIA assurance letter and other pertinent
documents.
N. ,
The week began with an entrance conference with the Division Director and senior staff.
Comprehensive interviews were conducted with key personnel on subject areas in addition to
extensive in-depth file reviews. At the conclusion of the week, an exit conference was held to
discuss findings with the Division Director and senior staff.
-------
CHAPTER 2 - GENERAL MANAGEMENT
2.1 BACKGROUND
The review team met with the Division Director; Associate Deputy Division Director;
Chief, Administrative Operations Staff; and the Extramural Management Specialist. The
purpose of this meeting was to obtain an understanding of Division management, staffing,
financial management, and extramural management issues.
2.2 STAFFING ISSUES
The inability to hire has plagued APPCD for many years. One secretarial position has
been filled in the last five years. It has been several years since a researcher has been hired and
although the contractor conversion has helped somewhat; it in no alleviates the loss of 16
personnel. APPCD showed 99 FTE's in 1996; with 1997 FTE's being reported at 85. The
disappearance of these 14 FTE's is very disturbing. At the same time APPCD was decreasing in
size, another office within ORD has more than doubled, throughout the "down-sizing and
reorganization".
The APPCD Division Director and his senior managers concur that among their critical
hires they list in-house technicians, research engineers and organic chemists.
2.2.1 MORALE
Low morale among the Division stems from lack of awards, negative empowerment, lack
of communication from ORD along with tighter restrictions being enforced. It is perceived by
APPCD that NRMRL senior management and ORD are not enthusiastically opposing the cutting
of FTE's, awards and funding that they (APPCD) deem necessary for .the mission of the Division
as well as the Laboratory.
2.3 FUNDING ISSUES
The Division looks at the historical usage of travel dollars along with prioritization. (1)
Project related travel, (2) Professional development and (3) generic trips. The Division Director
noted that there seems to be no clear rationale for travel dollars allocated to their Division
compared to that of the Immediate Office of NRMRL. (APPCD - 85 people -$178K/IO -30
people - S350K.)
Capital equipment >50K is excellent. Enables APPCD to keep state of the art equipment.
The purchasing of analytical equipment, which needs updating usually falls in the <50K dollar
category where considerable strain is felt. APPCD strongly recommends that pilot scale
equipment be considered Capital Equipment, whiefi is considered essential to their Pilot Plant in-
house research.
-------
Lack of PC&B dollars for awards continue to be a major concern among all staff
members of APPCD.
A constant source of irratation for the staff of the APPCD are the numerous safety
hazards they face. Several factors included are the age of the building combined with a leaky
roof and outdate plumbing compounded by the dwindling dollars received for Repair and
Improvement(R&I). A new building is in the budget of Congress but no target date for
completion has been established.
The review team conducted interviews with the Division Director, Associate Division
Director, the Chief, Administrative Operations Staff (AOS), the Internal Controls Coordinator
(ICC) and the Extramural Management Specialist(EMS), in an effort to determine whether the
Agency's Management Integrity Process was understood and being effectively implemented.
the Internal Controls Coordinator maintains all files and reference materials supporting
the Federal Managers' Financial Integrity Act (FMFIA). Provisions are outlined in EPA's June
6,1994, management integrity policy guidance; the Federal Managers Financial Integrity Act of
1982 (P.L. 97-255) and, Office of Management and Budget Circular A-123 (revised), dated June
21, 1995, on improving the accountability and effectiveness of Federal programs and operations
by establishing, assessing, correcting and reporting on management controls.
The Division Director holds meetings with his management staff (weekly), all hands
(periodically) and branch chiefs (bi-weekly). Information on policies and procedures is
distributed by the Chief, AOS to Branch Chiefs for dissemination to all staff. Senior managers
have performance^standards relating to accountability oh polices and procedures. The Division
went through the FMFIA vulnerability assessment several years ago, with no significant findings.
No recent assessments have been conducted. The Division completes the yearly assurance letter
and mid-year review. ,
The Division rarely adds additional guidance to those policies which originate from ORD
or the National Laboratory. Those policies and procedures are closely followed and monitored
by the Division Director. Accountability is built hi to the appropriate performance standards.
TQM committees worked on Timekeeping, Bankcards and Records Management areas a
few years ago and provided written reports td the Division Director. Based ori the review of
timekeeping, the training for timekeeping was updated.
The annual assurance letter is treated seriously, and the Division has reported several
accomplishments, such as Project Officer Training, quality assurance position conversions, and
an ADP hire. The Division did not identify any vulnerabilities or weaknesses requiring
corrective action.
The Division has experienced some difficulty with administrative/management support in
7
-------
the contracts area. Their non-ORD support is not as responsive as could be or very service
oriented. However, the small purchasing area support is good.
APPCD management and staff exhibit a strong recognition and understanding of the
importance of management controls. The Division has implemented and adhered to the controls
and the policies from which they derive. APPCD did indicate they would like more
communication in management areas between ORMA, the mega-lab and themselves.
2.5 ETHICS
The Ethics review was conducted during the General Management interview with the
main respondent being the Division Director.
y .
It was indicated that the Division Director serves as the Deputy Ethics Official (DEO).
All managers, supervisors and personnel involved in managing extramural instruments file a
Confidential Financial Disclosure Statement. The managers and supervisors have received ethics
training. Some of this training has been provided by the management analyst assigned ethics
duties. The Division does not currently have any staff serving as adjunct faculty.
APPCD has a DEO and is maintaining the required records.
2.6 DIVISION MANAGEMENT ISSUES
Concerns that APPCD expressed were as follows:
Would like to be consulted more on the service functions and issues that directly
affect the Division (i.e., HR services consolidation, extramural support changes). There
is a lack of communication to the,Division on these changes.
"Too much filtration and oversight" on acquisition and assistance processes.
More delegation needs to be given to the divisions.
There needs to be a tracking system for the extramural packages that must go
through NRMRL and ORMA.
The Division feels that the single responsibility center concept will result in
negative empowerment arid will lessen the ability to account for and track. 'This is
another area where communication is lacking on changes being made in the
administrative management area.
The Division Director indicated two extramural planning systems were
being utilized within the Division; the ORD Management Information
System (OMIS) with the Laboratory Implementation Plan System (LIPS) is
utilized along with a separate internal planning system of 1-pagers that is parallel
-------
to ONUS. The Director explained this was currently necessary to ensure the
consistency of date while flaws in ONUS are being corrected. Examples of the
internal planning system were provided to the Team Leader. The Director
indicated he was somewhat disappointed in the lack of fields in the OMIS/LIPS,
citing the Common Sense Initiative as an example. He expressed his opinion that
OMIS/LIPS needs to be more user friendly and comprehensive. The Director also
indicated the Extramural Management Specialist was involved in the planning
process.
-------
CHAPTER 3 - FINANCIAL MANAGEMENT
3.1 INTRODUCTION
The review team met with the Division Director; Associate Division Director; Chief,
Administrative Operations Staff, and the Extramural Management Specialist. The purpose of the
interview was to gain an understanding of APPCD 's financial management procedures and
processes. \
3.2 FUNDS CONTROL
3.2.1 INTRODUCTION
The review team interviewed APPCD's Funds Certifying Officer (FCO) to determine
their funds control procedures. The FCO was certified by the Director, ORMA after completing
the required ORD FCO training. The APPCD's Management Analyst serves as Alternate FCO.
Both FCOs report to the Chief, Administrative Operations Staff and have direct access to the
Division Director. Both FCOs are well trained and knowledgeable in Agency and ORD funds
control policies and procedures.
3.2.2 REVIEW OBSERVATIONS & FINDINGS
The FCO performs funds control functions for the entire Division which includes the
Office of Division Director, 5 Branches and the Administrative Operations Staff. The FCO
primarily monitors and tracks the expense and travel budget. The APPCD' s Environmental
Protection Specialist monitors and tracks the Extramural Budget.
The FCO maintains reference manuals that contain ORD transmittals, Agency policies
and procedures, Division delegations and other Federal policies on budget and finance. A review
of the selected funding documents showed that files are maintained in an orderly manner with all
necessary backup information readily available. The FCO has desk-procedures for all funds
control operations as required by Chapter 7, Section 6.6.2, ORD Policies and Procedures Manual.
'<'. '\
The Division uses planning documents to determine allocations to "tasks." Extramural
funds are tracked by the internal task number rather than the ORD Project Number. The
Division has one appropriation and five program elements that fund the extramural, expense and
reimbursable accounts. Funds availability is tracked using the ORD-3 Report, Division
spreadsheets and automated task funding reports. The Chief, AOS oversees funding allocations
and expenditures.
APPCD has an automated Document Control Register (DCR). The OCR's information
omitted Object Classes, Funding Descriptions and Project Number information. Funding
request transactions are entered into the DCR and IFMS on a daily basis. The expense and travel
10
-------
document control register is kept separately from the extramural register. The primary FCO
maintains the expense and travel register, while the Environmental Protection Specialist
maintains the extramural DCR. Obligations were not recorded in the extramural DCR. This
information is required to properly complete the monthly reconciliation process according to
Chapter 7, par. 6.8.1. Documentation supports the reconciliation process being performed but
with the missing information it is uncertain if all aspects are being reviewed.
Reconciliations are done on both registers on a minimum of a monthly basis.
Reconciliation is performed more often at the end of the fiscal year. It is performed manually
using IFMS and the DCRs. Corrections are sent to the appropriate finance office and reviewed
the next month to ensure completion. Modifications to commitments and/or travel obligations
must go through the same procedures as the original documents.
The Extramural funds are monitored and accounting information is assigned by the
Environmental Protection Specialist who is not a certified FCO. However all documents with
accounting information are reviewed and signed by the FCO. This arrangement could put the
FCO in a difficult situation for funds control certification. The Environmental Protection
Specialist is very knowledgeable in the Extramural funds control area, however Project Officer
duties preclude this individual from becoming an FCO. If personnel were available to shift the
Project Officer duties, the FCO duties in the Extramural area would be properly supported. The
FCO and the Environmental Protection Specialist have a close working relationship that does (
allow this situation to work in the Division's best interest.
V '
DIVISION RESPONSE: AOS did not feel that it was necessary to include the Object Classes
(OC) in the extramural Document Control Register (DCR) since the extramural expenditures
were limited to only a few OCs. When APPCD's internal data base was established for FY90,
we did not recognize the importance of including and tracking by, the ONUS project numbers in
the extramural DCR. In FY97 APPCD modified the extramural automated DCR to include the
object class and the OMIS project number for each commitment/obligation.
DIVISION RESPONSE: AOS does not receive all obligating documents to input in the DCR.
We will make every effort to obtain all obligating documents once we have reviewed the MARS
Report.
ACTION TAKEN: All obligations have been entered in the extramural DCR via award
documents and shall be kept up-to-date. -
3.2.3 RECOMMENDATIONS
3.2.3(a) The review team concluded that APPCD funds control functions are performed in
an organized manner and appropriate files are maintained. However, the review team notes that
the DCR had some discrepancies that should be corrected. The review team recommends the use
of OMIS which will support the project tracking that is needed to correct any DCR omissions.
With these corrections, the FCO operations will be in full compliance with established
procedures as stated in Chapter 7 of the ORD Policies and Procedures Manual.
11
-------
DIVISION RESPONSE: It is difficult from the Management Review Draft Report to know
what DCR had the discrepancies, therefore, both the intramural and extramural DCRs were
reviewed.
I it
ACTION TAKEN: Both the intramural and extramural DCRs have been checked fof any
discrepancies and, when necessary, corrections made.
DIVISION RESPONSE: The review team recommends the use of ONUS which would support
the project tracking that is needed to correct any DCR omissions.
ACTION TAKEN: We are continuing to input into ONUS, runjeports, and obtain training on all
modules of OMIS, which will help us is many different ways, including project tracking. In
FY96 we ran into so many barriers and unknowns that it was difficult to rely on the outputs of
the data. We also continued to run into problems that only the OMIS HOTLINE could resolve. I
believe that the staff is more aware of the capabilities of OMIS and plan to use it to its full
extent.
3.3 TRAVEL
3.3.1 INTRODUCTION
"v 'V .
APPCD's Management staff was interviewed regarding general travel management. At
the beginning of the fiscal year, Branch Chiefs prepare a travel plan which is reviewed and
prioritized according to Division procedures. Top priority is placed on project management.
Approximately 10% of the travel budget is reserved for emergencies. Accountability for; actual
usage is at the Branch level. Division Director monitors for appropriate use at the end of the year.
3.3.2 REVIEW OBSERVATIONS & FINDINGS
Branch Chiefs are the recommending official and the Chief, AOS is the approving official
for all Division Travel. Per the Resources Management Directives (Travel Manual, Chapter 2,
para 1), all travel approving officials are designated in writing. All international travel is
approved by Henry Longest. If changes occur before the trip is taken, an amendment is prepared.
If changes occur while in travel status, a statement is added to the Voucher for approval by the
Chief, AOS. If no changes occur, the Branch Chief may sign the voucher. The fimding for all
travel is recorded in the Document Control Register before the approving official signs. All
reference materials are kept in the Director's and Chief, AOS's office. IFMS and MARS manuals
are available to the Chief, AOS for report purposes.
Resources Management Directives - Travel Manual 2550B, Chapter 3.4 states that
frequent travelers should possess a government American Express card. APPCD employees use
these cards appropriately. RMD Chapter 2, para 2.b(9), requires vouchers to be prepared within
10 working days of the last day of the trip. Travelers are reminded verbally by Commitment
Clerk for Domestic Travel and for International Travel by the Administrative Officer. Division
Director feels that the American Express payment process assures prompt vouchering.
12
-------
Local travel and parking fees are approved if destination is under 50 mile radius from EPA
site. The Division has no Site Specific Superfund Travel. International Travel policies include
the preparation of an ITP, and a Trip Report upon completion of trip. The Division's international
travel is primarily reimbursement travel. The Division occasionally issues invitational travel
authorizations to interviewees or peer reviewers. These are done rarely. There is no government
contractor travel. :•
DIVISION RESPONSE: The report states: "The funding for all travel is recorded in the
Document Control Register before the approving official signs." This is incorrect. The approving
official signs all TAs prior to recording in the DCR.
FILE REVIEWS
File reviews included five local travel reimbursement requests, five domestic travel
authorizations and subsequent vouchers, one invitational travel authorization, one international
travel authorization and voucher, and two reimbursable (one international) travel authorizations.
3.3.2.1 LOCAL TRAVEL
Local travel reimbursement vouchers included one for a private plane - used to take aerial
photographs. Most local travel was authorized for attending or teaching training courses and for
trips related to bankcard purchases. All reimbursements were properly documented and records
kept appropriately. Per Diem and lodging are not included in local travel reimbursements per
EPA Comptroller Policy Announcement Nos. 89-02,2/27/89, and 89-02 Amendment 1,11/21/89.
EPA Comptroller Policy Announcement No. 89-02,2/27/89, also requires that each Finance
Office define the cities, towns, and counties which are included in the 50-mile radius for the
organizations to use as guidelines for local travel reimbursement areas. This document is located
in the Division Administrative Officer's office.
3.3.2.2 DOMESTIC TRAVEL
Authorizations were prepared correctly. Some blocks on the TA were left blank such as
the "Applicable Regulations" block - This omittance is common and the Finance Office does not
enforce the requirement. Traveler's information, purpose codes, SFO's and funding blocks were
appropriately completed. Some signature dates did not appear on xeroxed TA and Vouchers -
commonly due to making the copy before signature and forwarding the original signed document
to Finance without an additional copy made with signatures. Therefore, it was difficult at times
for the review team to discern whether travel was approved before the trip began. However,
further research by the review team secured signature documents from the FCO which verified
prior approval was obtained.
Copies of receipts were not available and apparently are not kept at the Commitment Clerk
level. Copies of receipts are kept primarily by the Traveler. (The regulations do not state that
copies of receipts must be kept.) '. v
13
-------
One authorization showed that the POV was not authorized in the Remarks section of the
TA as required by Resources Management Directive (RMD) 2550B Exhibit 3-3 and Item 12 in
RMD Chap 4, para 5.h. Funding for POV is listed but POV statement needs to be authorized as
advantageous to the government - even if only for trip to and from airport. The review team
recognized this as an isolated incident and therefore no finding was recorded.
The "Advance" block is out of date in that ATM advances are not "signed" for on the TA.
Therefore there is no finding regarding advances not being signed for on the TA.
3.3.2.3 REIMBURSABLE TRAVEL
A
-j ''
One TA did not show a checkmark in the "Type of Travel" block. One voucher for
reimbursable travel took 37 days to submit as opposed to the required 10 days.
3.3.2.4 INTERNATIONAL/REIMBURSABLE TRAVEL
Henry Longest's approval is not dated, Increased per diem rate is correctly amended and
approved on the voucher. Per Diem expenses are properly converted from foreign currency to the
US dollar as appropriate.
3.3.2.5 INTERNATIONAL TRAVEL
Henry Longest's signature is again not dated. The International Trip Report is not dated
(there actually is no block for a date of preparation). The ITP is prepared in accordance with
ORD Chapter 7, para 6.9.1.
3.3.3 RECOMMENDATIONS
Only a few minor suggestions are offered since APPCD is in compliance with regulations.
3.3.3(a) Copies of receipts would be helpful supporting documentation - either at the traveler or
commitment clerk level. This is probably occurring at the traveler level - but the file review was
at the commitment clerk level so invoice copies were not available.
3.3.3(b) Dates of signatures should be checked to assure proper tracking.
/
3.3.3(c) Per RMD 2550B Exhibit 3-3, and RMD Chapter 4, para 5.h) authorization of POV and
rental car must be in the Remarks block of the TA - not just in the funding blocks.
DIVISION RESPONSE (a): AOS contacted the branch's support staff and determined that
their files contained copies of receipts to support travel documents. It is not the responsibility of
the FCO to keep receipts.
NO ACTION NECESSARY
14
-------
DIVISION RESPONSE (b): The TA does not require dates of signatures but this is a
requirement for vouchers.
ACTION TAKEN: AOS sent out a memo to recommending and approving officials to ensure
signatures on TAs and vouchers are dated. (See Attachment 1)
DIVISION RESPONSE (c): To authorize use of POV as the primary mode of transportation
does require a special justification that proves it is more advantageous to the government.
However, there is an exception. According to Federal Travel Regulation 41 CFR Part 301-4.2, a
special justification is not required for round-trip by POV in lieu of taxi between common carrier
terminal and office or residence. Our local travel office and travel training we have received,
support this position. We were instructed that the 'X' on the POV line in Block #11 is sufficient
without providing a justification in the Remarks block. We verified this information with GSA.
NO ACTION NECESSARY
3.4 TIMEKEEPING
3.4.1 INTRODUCTION
The Division's timekeeping process was reviewed to determine if Agency procedures are
being appropriately and effectively implemented. The review team met with the Division
Director and Chief, AOS to get a general overview of APPCD's process.
Included in interviews by the Review Team were four Timekeepers, four Supervisors, and
two Designated Agents. Time cards studied were Pay Periods 26/96 (9/15-28/96) and 02/97
(10/13-26/97).
3.4.2 REVIEW OBSERVATIONS AND FINDINGS
3.4.2.1 PROCESS
APPCD timekeepers have a listing of their employees' daily schedules as reference
material. (Chapter 2, para 7a of the Timekeeping Manual requires timekeepers to verify the
documentation on the Timecard.) Flex time is utilized by most of the staff.
Employees are trusted to show up on time and leave on time unless leave is documented.
All timekeepers mentioned that their group was small and manageable enough that a formal
system of "logging in or out" or "watching" was definitely not necessary. One timekeeper
mentioned that her schedule of starting work at 6:30 am allows her the opportunity to periodically
check on attendance for the morning hours.
Sick leave is usually requested as the need arises (AM phone call requesting sick leave
from appropriate Division personnel) but extended sick leave or annual leave is requested using
a Form SF-71. Family Leave is still fairly new to this Division but on occasion is used.
Timekeepers were reminded that this type of leave needs special notation and supervisory
approval just like comp time.
15
-------
The supporting documents are kept by the timekeeper for use in the completion of the
Timecard and are attached to the cards at the time of submittal to the supervisor for approval and
signature. Time cards are completed in the appropriate time frame and submitted for approval to
the supervisor, (which is required by Chapter 2, para. T.b of the timekeeping Manual) who
reviews, signs, copies and seals in an envelope and hand carries to the Designated Agent.
3.4.2.2 RECORDKEEPING i
Timekeepers interviewed kept copies of cards even longer than necessary (Chapter 2, para
7.a.5 requires cards be kept 90 days). Supporting documents are required to be kept for three
years, (above reference in the manual states, " three years for records supporting entries on the
Time cards.") since they are actually originals. APPCD timekeepers' are keeping them
appropriately. *
3.4.2.3 TRAINING
All timekeepers receive training when it is offered from their Finance Office. Most
changes in policy are communicated by E-mail or memorandum. Timekeepers felt they had
appropriate knowledge, reference materials, and training for this function. (EPA Assistant
Administrator Memo regarding Time and Attendance Reporting, Dated May 4, 1992, requires that
timekeepers attend the timekeeping training programs offered).
3.4.2.4 SUPERVISORY APPROVALS
The Supervisors interviewed indicate they receive Time cards and supporting leave and
comp time documentation for their employees directly from the Timekeeper. Two Supervisors
keep their own logs showing leave taken and changes in schedule, and use this log to check the
accuracy of the timecards.
The Division Director scans the cards and the Time and Attendance Verification Report
and if anything looks odd, requests more documentation. He does not make the copy of his
signed cards. His secretary does and then seals and hand carries to the DA. One Supervisor
indicated that he seldom approves comp time but in general, when comp time accumulates over
60 hours, the Division Director must approve.
After signing the cards, the Supervisor copies the cards (cc is for Timekeeper), seals them
in an envelope and, hand carries to Designated Agent (Chapter 2, para 8 requires hand carrying of
cards directly to the Designated Agent). After the pay period, the Time and Attendance
Verification Report is reviewed, spot checked, and kept on file for two months. Any
discrepancies are reported to the Payroll Section by the Timekeeper (per Chapter 2, para. 7c).
Proper disposal of this report may occur anytime thereafter.
16
-------
3.4.2.5 DESIGNATED AGENT fDA)
The Designated Agent receives the cards in sealed envelopes from the Supervisor per
Chapter 2, para. 8.a of the Timekeeping Manual). She sorts and spot checks for totals, checks for
signatures and submits them to Finance. Cards are submitted to the servicing finance office in a
sealed envelope (batched by SSN) as per Chapter 2, par 8.a(l) through (2) of the Timekeeping
Manual. She keeps copies of cards in a locked cabinet. The Alternate DA performed this duty
three times last fiscal year and is using similar procedures. Timecards for the next pay period are
distributed by the D.A. to the applicable Timekeepers per Chapter 2, para 2.d of the Timekeeping
Manual. •
• •f
3.4.3 RECOMMENDATIONS
l
3.4.3(a) Interviews left some doubt as to consistency in the copying of the completed, signed
timecards. The Supervisor should make the copy and hand carry the original to Designated
Agent. Some Supervisors chose to have a Secretary (SEE or Fed but not the timekeeper) make
the copies and then the Secretary hand carried the cards to the DA. Chapter 2, para 8.a of the
Manual states that Supervisors are required to seal the cards in an envelope and carry directly to
the DA. ,
3.4.3(b) Interviewees were questioned as to any improvements they would like to suggest. Most
indicated that electronic timekeeping would be better (although some reservations regarding
security and approvals exist). One Timekeeper felt that the new "exception reporting" was
"silly". If true streamlining is to be accomplished, putting in less numbers on the card was not
very efficient. She felt it makes more sense to just put "80" in the total column and not worry
about the codes.
3.4.3(c) Division Director indicated considerable confusion as to appropriate policy during
"furlough". He indicated this type of confusion may be alleviated with computerized
timekeeping.
DIVISION RESPONSE A: Timekeepers' copies of T&As do not include the supervisors'
signature unless the supervisor makes a copy for the timekeeper before hand delivering to the DA.
ACTION TAKEN: The DA has offered to the supervisors to make copies of the signed
timecards for the timekeepers if they request. (See Attachment 2)
B. NO RESPONSE OR ACTION NECESSARY.
I
C. NO RESPONSE OR ACTION NECESSARY.
3.5 BANKCARD
3.5.1 INTRODUCTION
17
-------
Interviews were conducted with the Division Director and the Chief, AOS regarding
general management and accountability issues related to the Bankcard Program. Questions were
asked in reference to the Division's staff carrying out management responsibilities, including
compliance with laws, regulations, policies and procedures.
Mini Total Quality Management (TQM) teams were formed to address bankcard issues
relating to the Federal Managers Financiallntegrity Act (FMFIA) (See attachment section in
manual.) These teams met numerous times to address Description/Implementation of the
System; Vulnerability/Concerns/Risks; Lack of Lab Policy and Procedures for Administering the
Program; Audits; Restriction of the number of cardholders; Cardholder Training and
Accountability and Written Bankcard Policy and Procedures. It is clear that Management is very
involved and dedicated to procedures in the management integrity process.
In addition to the above interviews, the Review Team also met with four Approving
Officials and four Bankcard Holders.
/, '
3.5.2. BANKCARD HOLDERS - REVIEW OBSERVATIONS & FINDINGS
3.5.2.1 FILE MAINTENANCE & DELEGATION OF AUTHORITY
Documentation Required by Paragraph G.5 of the EPA Guidelines For Use of U.S.
Government Bankcards, dated March 1993. (Please Note: EPA Guidelines for Use of U.S.
Government Bankcards, dated August 1996; was unavailable for the Team's reference during this
management review. However, after review of the new guidelines, the observations and findings
remain the same.) ,
APPCDs' Bankcard Holders reviewed had submitted the proper Delegation of Authority
memorandum issued to them from Headquarters Procurement Operations Division. This
authority clearly stated the name, single purchase lirm't, office limit and 30-day limit for the
Bankcard Holder and their assigned Approving Official.
Further, the Bankcard Holders all maintained copies of their Bankcard Logs, Reconciled
Statement of Accounts, and Purchase receipts for the monthly billing cycles for at least the last 12
months, and in some cases, records were retained for a longer period of time.
3.5.2.2 PROCUREMENT REQUEST
FINDING
Bankcard Holders DID NOT have approved Procurement Requests for Bankcard
purchases other than ADP purchases. This is required in the EPA Guidelines For Use Of U.S.
Government Bankcards, March 1993, (Pg, 28, G., 2, (b)).
RFfYllVTMFNDATION
18
-------
3.5.2.2(a) Bankcard Holders are responsible for acquiring signature bv the Approvine Official
for each individual item on log sheet or prepared procurement request prinr tn purchase.
\
DIVISION RESPONSE: Based on required training for bankcard holders and approving
officials, it is not required to acquire signatures by the Approving Official for each individual item
on the log sheet or on a Procurement Request (EPA 1900-8). Signature on the log sheet at the end
of the cycle is sufficient. AOS confirmed this with Alan Rirter, the Bankcard Program Trainer.
(See Attachment 3)
NO ACTION NECESSARY
N | , /
3.5.2.3 RECONCILED STATEMENT OF ACCOUNT
, ' \
Bankcard Logs and Statements of accounts were reviewed for the last completed 30 day
cycle, August 18,1996 through September 17,1996 and the previous 6 month completed billing
cycle. January 18,1996 through September 17,1996 for all 4 Bankcard Holders.
EPA Guidelines (Section G, par. 3.a.3, page 29) state, "sign and date each page"; of the
Statement of Account. The Bankcard Holder and the Approving Official did not have copies of
the front and back page of this statement. Therefore, the review team was unable to verify
certification of items ordered, received, or that charges included on the statement were correct
and approved for payment. This missing back page also made it impossible to verify submission
of the Statement of Account to the Approving Official for review within the 5 working days of
receipt from the Contracting Bank (EPA Guidelines, Section G, par. 3.a.5, page 29),
This also applied to the Approving Official's submission to the Cincinnati Financial
Management Center within the required 5 working days of the receipt from the Bankcard Holder
(EPA Guidelines, Section G, par. 3.b.3, page 30). (Note: The FCO did produce a copy of these
statements so we could verify signature. It is recommended that no finding be written for this
error, only poor records management.) <. , '
The Bankcard Holders did include a brief description for each Bankcard charge listed on
the statement. (Review Team Recommendation: Make these descriptions brief BUT
complete.) Bankcard Log cycles were also annotated on the statement (EPA Guidelines, Section
G, par. 3.a, page 29-30).
Prohibited Items purchased on the Bankcard: Portable Electric Power Tools were listed on
one Bankcard log sheet, the review team was unable to find an invoice, or order date received,
etc. to verify "what this item was" and to see if it was on the prohibited list.
(Review Team again recommends that the descriptions on the log sheet be brief BUT
complete.). \
There was one Bankcard log sheet which showed furniture being purchased from a vendor
other than the required "Unicor" from the Federal Prison Industries, Inc. If another vendor is
used, a waiver must be submitted and kept with the file. (Note: However, during the same billing
•,'.*••
19
-------
cycle, this furniture was detected as being prohibited, and was promptly returned.) The
UNICOR-QU1CK SHIP (catalog only) is the vendor that Bankcard purchasers are authorized to
use. Bankcard Holders are still required to obtain approval of the, Ipcal property officer prior to
making any purchase from Unicor with the Bankcard.
There were no disputed charges to verify.
DIVISION RESPONSE: Bankcard Holders or Approving Official did not have copies of the
back page of the statement with the Approving Officials dated signature. The Team
recommended that copies of this page be copied after signatures.
ACTION TO BE TAKEN: AOS will send out a memo recommending that the Approving
Officials make a copy of the back page of the statement to be kept by the Bankcard Holder.
f '
DIVISION RESPONSE: The Review Team recommends that the log sheets and statements
include a COMPLETE but brief description of item(s).
ACTION TO BE TAKEN: AOS will send out a memo with the Review Team's
recommendation.
DIVISION RESPONSE: The Review Team indicated that portable electric power tools are
listed as prohibited items.
ACTION TAKEN: AOS contacted the Bankcard Program Coordinator and this item was
removed from the restricted list in 1995. (See Attachment 3)
3.5.2.4 SUPPORTING RECEIPTS
/
Copies of customer receipts from Bankcard purchases were found in 3 out of 4 Bankcard
Holders files. However, several receipts shows Sales Tax being charged, (EPA Guidelines,
Section K, 5, page 35) "Bankcard purchases are TAX EXEMPT. It is IMPERATIVE that the
Bankcard Holder remind the merchant before the purchase is made that the purchase is TAX
EXEMPT". (The Review Team did not feel this warranted a Finding but strongly suggest this
rule he followed more closely.}
All Bankcard Holders are in compliance with the EPA Guidelines, Section D, par. 10,
"Bankcard Holders shall not split requirements in order to stay within their single purchase limits.
It was noted that in the Items Purchased Column, the items are referred to as Laboratory or Office
supplies, this would make it difficult to see a "split" if you can't determine exactly what it is that
was purchased. (Again, the review team recommends that a brief, but more complete
description be given.) ,.
DIVISION RESPONSE: The Review Team suggests that the Bankcard Holders remind the
merchant, before the purchase is made, that the purchase is TAX EXEMPT.
ACTION TO BE TAKEN: AOS will send out a memo reminding Bankcard Holders that all
20
-------
purchases are tax exempt.
3.5.2.5 BANKCARDLOG
APPCD Bankcard Holders used the Bankcard log to record monthly Bankcard purchases.
EPA Guidelines, Section G, par.2.a, page 28 required the "entire" log be completed. The
reviewers found item 4, Date Ordered, and item II, Date Received not always complete. The
supplies and services to be purchased must be readily available in order to be received prior to the
end of the billing cycle in which the order is placed, EPA Guidelines, Section D. Basic Features,
7, page 21. This could not be easily determined. Item 5, Items Purchased, this description was
too vague, which made it hard for the reviewers to reconcile the log sheet with the statement of
account.
All Bankcard Holders had the proper signatures required on the Bankcard Lpg Sheet.
(EPA Guidelines, Section G, par. 2.c, page 27). The Funds Certifying Officers records were
accurate with the appropriate accounting information.. All logsheets were certified by the FCO
and sent to the Cincinnati Financial Management Office in the required time frame. (EPA
Guidelines, Section G., par 2.c, page 28).
i
The Docuirient Control Numbers and their corresponding commitment amounts matched
the FCO's Document Control Register. (NOTE: One bankcard holder kept a DCR for their
Branches expenditures (bankcard as well as PR's)). The document control register (FCO) was
quite long, due to the fact that each line item on the Log Sheet has its own document control
number. The review team discussed this procedure with the FCO and Chief, AOS. Cincinnati's
procedures may prove to be more efficient. APPCD will pursue.
DIVISION RESPONSE: the Review Team noted that the "Date Ordered" and "Date
Received" blocks on the bankcard log sheets were not always complete.
ACTION TO BE TAKEN: AOS will send out a memo reminding Bankcard Holders to submit a
complete log sheet including the date ordered block and date received block, if item(s) have been
received prior to the end of the billing period.
3.5.2.6 SUGGESTIONS FOR IMPROVEMENT BY THE BANKCARD HOLDER
Vendor codes need to be re-examined.
Would like to have a central person for Bankcard. Too time consuming for current staff.
Reduce regulations.
Less approvals for smaller items.
Reduce prior approval for smaller items.
Allow Bankcard Holders to carry their visa's with them.
21
-------
3.5.3 SUMMARY /RFrnMMENDATIONS
The purpose of interviewing and conducting a file review on the Bankcard Holders was to
gain a knowledge of the Bankcard Holders understanding of their responsibilities in the
application of policies and procedures of the Bankcard Program and how it relates to their
organization.
It was obvious during the interviews that APPCDs' Bankcard Holders clearly understood
their single purchase limits, 30-day spending limits, who is authorized to use their cards, storage
of Bankcards when not being use, and the Appropriation Restructuring Implementation
Guidebook Release #94-02. File Maintenance was an area in which most Bankcard Holders had
questions. They were aware of and did have a copy of the Delegation of Procurement Authority
retained in their permanent file.
Purchase Requisitions were not obtained for purchases other than ADP. EPA Guidelines
Section G, 2.(a) states, "Prior to using the Bankcard, it is the responsibility of the Bankcard
Holder to ensure that sufficient funds are available to cover any purchases in order to prevent an
unauthorized procurement". Without this advance PR, it was impossible to determine that funds
were allocated or that funds were authorized.
Copies of the Bankcard Logs were maintained by the Bankcard Holder however, copies of
authorized signatures by the approving officials or the FGO were not maintained. Copies of these
were found with the Funds Certifying Official.
Copies of Bankcard Purchase receipts were not always kept with copies of the Monthly
Bankcard Statements. It was difficult to determine how the bankcard logsheets and the monthly
statements were reconciled. Most interviewed Bankcard Holders were unaware that they must
maintain copies of the completed Statements of Accounts with ALL documentation for at least
one full year. Most Bankcard Holders kept documentation from day one. Bankcard Holders were
excellent when adhering to the required timeframes for submission of Bankcard Logsheets to their
Approving Officials. With limited documentation, the review team was unable to determine as to
when the FCO signed and forwarded the Bankcard Logsheets to the Cincinnati Financial
Management Center (CFMC).
The Statement of Accounts (each charge) were annotated with descriptive information as
well as to which Logsheet they referred to, however; again, the review team recommend that it
would be most helpful if the descriptions cpuld be more detailed, i.e.,....office supplies, laboratory
supplies. ,
Statement of Accounts were copied but, only the front page. Each Bankcard Holder must
certify on their individual Statement of Account that the item(s) ordered have been received; and
the charges are correct for payment. This signature is located on the back side of the statement.
The team was unable to ascertain when the Statement of Account is forwarded to the Approving
Official for review. EPA Guidelines, Section g, 3(5), page 29, states it must be forwarded to
22
-------
Approving Official within 5 working days after receipt.
f
3.5.3.(a) // is recommended tins statement be copied front and back by hnth the Rankcard
Holder and the Approving Official A Time/Stamp machine mav be useful far losing in the
Statements.
DIVISION RESPONSE: The Review Team was unable to obtain copies of authorized
signatures by the approving officials or the FCO on the Bankcard Logs.
ACTION TAKEN: The FCO sends the Bankcard Holder a copy of his/her log sheet with
signatures and accounting data.
DIVISION RESPONSE: Bankcard Holders were not aware that ALL documentation must be
kept for at least one full year.
ACTION TO BE TAKEN: AOS will send out a reminder regarding the length of bankcard
documentation retention.
DIVISION RESPONSE: The Review Team could not determine whether the statement was
submitted to the Cincinnati FMC within the 5 day period and suggested that the statement be date
stamped when received by the Bankcard Holder. This is not a requirement.
ACTION TO BE TAKEN: AOS will send out a memo with this suggestion.
3.5.4 REVIEW OBSERVATIONS AND FINDINGS -APPROVING OFFICIALS
3.5.4.1 - DESIGNATION OF APPROVING OFFICIAL
APPCDs' Approving Officials had the proper Designation of Approving Official
memorandum issued to them from Headquarters Procurement Operations Division. This
authority clearly states the Approving Official Account Number, the 30-day Approving Official
office limit and the names and spending limits for the Bankcard Holders assigned to them.
3.5.4.2 TRAINING OF BANKCARD HOLDERS
All Approving Officials stated their Bankcard Holders have received the required training
by attending the "Bankcard and Call Ordering Officers Course". Follow-up refresher courses are
also taken on as given basis. (RTP had a refresher course during the time of Management Review)
3.5.4.3 PURCHASE REQUESTS. AUTHORIZATION. RECORDING
The policies and procedures in place for the use and control of bankcards are varied. Most
prior approvals are obtained "verbally". Only 1 out of the 4 approving officials required their
Bankcard Holders to fill out a purchase requisition prior to making the purchase. There is an
exception, ALL ADP procurement were made with a prior purchase requisitions by ALL
Approving Officials.. Usually the Approving Official signed the Bankcard Logsheet AFTER the
23
-------
billing cycle. Again, one Approving Official kept his own "internal tracking system" to verify
purchases made on the logsheet. It wasn't clear as to how the remaining three Approving
Officials verified that the purchases were valid. See Review Record of Finding and
Recommendations (FAR) under BANKCARD HOLDERS, PROCUREMENT REQUEST. (EPA
Guidelines, G.2,(b) Page 28.
3.5.4.4 RECONCILIATION OF BANKCARD STATEMENT
One approving official actually verified the statement with his own copies of prior signed
purchase requisitions. It was found that only one approving official kept copies of the signed
statement of account. Most records were kept with the Bankcard Holder. After reviewing the
Bankcard Holders files, it was noted that the Bankcard Holder only copied the front page of the
statement. It was difficult to determine if the correct signatures or dates were made within the
EPA Guidelines Section G. Par.3.a.3 page 29. See BANKCARD HOLDERS, RECONCILED
STATEMENT OF ACCOUNTS
; /'
3.5.4.S PERFORMANCE STANDARDS
(
All Approving Officials have performance standards for the Bankcards Holders as well as
standards for themselves.
3.5.5 SUGGESTIONS FOR IMPROVEMENTS BY APPROVING OFFICIALS
1 , /
1. E-Mail from Headquarters (Stacey Brown and Karen Lee) is not always received
by the Bankcard Holders and Approving Officials. This needs^to be improved.
2. EMS should have warrant over $2,500.
3. Bankcard Holders need to have more authorization authority. Technicians need
some way of blanket authority to order widgets.
4. Approving Official would like a central point for Bankcard purchases. Ideally, 1
person full-time.
5. Raise the limit on single purchases, ADP, Lab supplies.
3.5.5 RECOMMENDATIONS
The purpose of interviewing the Approving official was to gain a knowledge of how the
Bankcard purchase systems works within his/her Branch. Approving Officials relied heavily
upon the Bankcards Holders to do the bookkeeping.
Areas needing improvement were the signature sign-offs on each and every purchase
request (not just ADP). The Approving Official must approve all individual purchases and must
indicate the approval on a PR or on the Bankcard Log. Three Approving Officials logsheets were
24
-------
not reviewed with any PR or other form of spreadsheet tracking system prior to signature sign-off.
The Approving official must review the Log at the end of the Bankcard 30-day billing cycle and
after approval, sign, date and immediately forward the log tq the F, unds Certifying Official (FCO).
(EPA Guideline G.2, page 28-29)
The Statements of Accounts were not kept by the Approving Officials. The copies were
kept with the Bankcard Holders (one Approving Official did have), therefore, the signatures were
not a matter of record in their files. By not having these copies (front and back) it was difficult to
see if the Approving Official approved purchases on this statement ,and if the approvals were
done in the time frame outlined in (EPA Guidelines, Section G, 3.3, page 30) The Approving
Official must review, approve and ensure that the original Statement of Account is properly
annotated, signed and forwarded to the Cincinnati Financial Management Center within 5
working days of the receipt of the Statement of Account from the Bankcard Holder. The
reviewers found signatures other than the approving official, it was documented that this person
was serving in an acting capacity (memo on file). Proper signature was obtained.
25
-------
CHAPTER 4 - EXTRAMURAL MANAGEMENT
4.1 EXTRAMURAL MANAGEMENT GENERAL
The Team Leader of the review team assigned responsibility for reviewing extramural
vehicles and participated in the entrance conference while the remainder of the team initiated the
file review process. During the entrance conference, extramural management was discussed with
the Division Director and appropriate staff, including the Extramural Management Specialist
(EMS), using the ORD Management Review Program interview guides.
'\:
Prior to the start of the review, a stratified sample of representative extramural vehicles
was selected by the Team Leader for the extramural review team from the list generated from the
ORD Management Information System (OMIS) on 09/23/96. Vehicles that had expired by the
time of the review or that were due to expire during the first quarter of FY 97 were not included
in the review. In addition, those selected had been in effect for at least one year so as to provide
sufficient opportunity for the review team to evaluate post-award activities under the vehicles.
While purchase orders were listed in OMIS, they were not considered part of the pool of
contracts.
The review team conducted reviews of seventeen of the forty-five active cooperative
agreements (CAs), nine of the twenty-nine active interagency agreements (lAGs), and seven of
the twenty-one active contracts. In addition, three Work Assignment Manager's (WAM) files
associated with Contract 68-D4-0005 were reviewed as was the Delivery Order Project Officer's
(DOPO) file under the FAIR (ADP support) contract.. The review team also interviewed seven
CA Project Officers (POs), six IAG POs, five contract POs, and three WAMs. The specific files
reviewed and personnel interviewed are identified later. '
The review team found the majority of the Division's staff involved in the award and
management of extramural agreements to be knowledgeable of their roles and responsibilities.
All personnel within the Division were cooperative.
The review team found that the Division staff rely on the EMS for support and advice.
POs acknowledged the EMS as a resource, noting that he provides base level training on a
regular basis and is always available for specific questions and concerns. They indicated he often
acts as an intermediary with the Grants Administration Division (GAD) or Contracts
Management Division (CMD) to provide a translation between the needs of the scientists and the
limitations of law and regulation. They were confident that he would notify them of any major
changes in policies and procedures and that he would maintain an up-to-date library of
acquisition and assistance policies and procedures for their use. The review team also found that
Division managers rely upon the EMS to provide expert advice and analysis relative to
extramural management issues.
At the exit conference attended by the Division Director, Branch Chiefs, the EMS, and
other appropriate staff; each member of the. review team presented preliminary findings related to
issues, both positive and negative, which are further developed in the following sections of this
26
-------
report. The Extramural Team Leader stressed that the primary client, was the Division's
management with the goal of providing the Division Director with an independent and expert
analysis of the strengths and weaknesses of the extramural management program.
4.2 CROSS-CUTTING ISSUES
INTRODUCTION. The review team identified a number of cross-cutting issues that related to
more than one type of extramural instrument, discussed as follows:
1. Files Do Not Consistently Reflect Appropriate Review and Approvals.
Findings
' \f ' •
The review team identified a variety of instances in which the files do not clearly
indicate that appropriate approvals were requested, or that they were received, if requested. In
some instances, it is likely mat the approvals were in fact requested and given, but the file simply
lacks the documentation. In other cases it appears that in fact the approvals were overlooked.
Note that the review team is not including here the problem of the absence of signed copies of
approvals from personnel outside the Division. It is common in many labs that the PO has a file
copy of a request as it was sent to higher authority for approvals, but not a copy of the actual
approval. This is because the file was sent from the approval official directly to GAD or the
Office of Acquisition Management (OAM), not back to the PQ. The review team does not ,
consider this to be a significant weakness. If the file contains evidence both of submission of the
request and of action by GAD or OAM, the review team assumed that the intervening officials
approved. It should be noted that several of the files reviewed contained all required
documentation and that the documentation was appropriately approved. Subsequent to the
review, many missing documents were located by the APPCD EMS, and this report has been
revised accordingly. While that helped to relieve concern about whether the Division actually
complied with the rules, the review team notes that the poor files caused a great deal of time to
be wasted.
A. Examples in the "undocumented" area include:
o IAG DW80936395 with NASA was written by the Office of Air Quality Planning and
Standards (OAQPS) and transferred to NRMRL a year later. OAQPS did not provide the award
file, so there is no record of any of the pre-award approvals. Because this was written by an
organization outside ORD, the file will not, and need not, contain any ORD specific approvals;
however the review team believes the project should have been reviewed for QA and potential
environmental impact, and in any event the file should include evidence of what reviews and
approvals were obtained by OAQPS or at least show ORD efforts to obtain the documentation.
ORD is responsible for the overall acceptability of any agreement it manages, and the file should
demonstrate an effort to determine the acceptability of the agreement as of the time it was
transferred to ORD.
o LAG DW96936393 with the Corps of Engineers (COE) contains inadequate evidence
27
-------
of APPCD having obtained required Senior Resource Official (SRO) approval of the initial
award. Subsequent to the review, the EMS provided the review team with a copy of an unsigned,
undated decision memorandum that was routed from the Division Director to the SRO. Later
\ 41'
amendments do reflect proper review, so the team feels this is a file documentation issue, not a
substantive failure.
o CAs CR822871 and CR822870, both with RTI, contain evidence of coordination with
the SRO of AA during the presolicitation phase, but do not indicate that the actual award
packages were approved appropriately by ORD. These findings were included in the
undocumented category because the review team expects that the documented involvement at the
planning phase indicates it is likely that the final packages were properly reviewed; but again,
this cannot be determined from the file.
o During the on-site review the review team noted most of the POs' files lacked
documentation relative to the review of Quality Assurance Plans by the APPCD Quality
Assurance Officer (QAO). As a result, the review team initially concluded the QA reviews had
not been conducted. Subsequently the APPCD EMS secured documentation from the QAO
which documents the reviews that were conducted or which explained why a QA review was not
conducted. Some examples of the latter include:
—CA CR823020 with Georgia Tech did not include any environmental measurements
during the first two years of the agreement. A QA Plan is now due in March, 1997.
--The APPCD QA Officer delegated the QA review under IAG DW 80936395 to NASA
since the majority of the work was performed and managed by NASA's contractor.
The review team believes that the POs' files should include documentation indicating QA
reviews that were conducted or waived, as the case may be. Accordingly this finding is included
in the "undocumented" area even though the QAO did retain documentation of reviews she
conducted. Note that currently APPCD requires written approval of QA Project Plans for
Categories I, H and HI type projects by both the QAO and the PO. For Category IV, the PO
alone approves the plan and forwards it to the QAO. Files need to be documented relative to
these required reviews.
i
One of the main contributing factors to these cases is that the affected files generally were
very poorly organized. It appears that the POs involved did not assemble a complete file and
organize it in a manner that would effectively demonstrate the management of the process. The
files generally were older, and the POs may not have had a good understanding of their
responsibility for award requirements and of filing needs. Well organized files do not necessarily
assure proper award and management of extramural agreements, but poor files often contribute to
real or apparent errors or omissions.
\
B. Examples in the "overlooked" area include:
o Three examples in the "overlooked" area are found in the duct cleaning work
28
-------
assignment under Contract 68-D4-0005 with Acurex.
1. There was no evidence that the requirement for OMB approval of the administration of
the questionnaire was obtained or recognized. It appears from the interviews that the
researchers thought the OMB clearance would not be needed if the ultimate study
involved 9 or fewer homes.
2. The data collected appears to constitute a "system of records", subject to the Privacy
Act. Steps were taken to protect the data, but the various publication requirements of the
Act do not appear to have been recognized or complied with.
3. Finally, as to that work assignment, the review team believes that the study should
have been reviewed and approved in accordance with the EPA Order on Human Subjects
(1000.17A), or 40 CFR Part 26. The research protocol called for cleaning the HVAC
ducts in the homes of the subjects and testing the air in the homes befdre and after
cleaning to identify and measure the levels of dust and biological contaminants. It was
recognized that there would be some elevation in contaminants, and there should at least
have been a documented decision of whether this constituted manipulation of the
subject's environment. It is not clear whether any "private information" was collected
that would have been subject to the regulation or Order.
o A work plan submitted under CA CR823052 with RTI indicated the intent to seek
survey information from a large number of sources, but there was no indication OMB clearance
was sought or obtained. r
o IAG RW89936334 with DOE was sent directly from NRMRL to GAD without
indication of approval or routing through the SRO/AA Avhich was, at the time of processing,
required for all funds-in lAGs. Similarly, CA CR823052 with RTI should have had SRO
review, but the file indicates direct transmittal frbm APPCD to GAD.
The lack of documentation relative to reviews and approvals as stipulated in the Agency's
and/or ORD's policies and procedures, if nothing else, calls into question whether the Division
did obtain the required approvals. If indeed the approvals were not obtained, then the Division
can be cited for a lack of sound management controls. By failing to obtain approvals, the
Division prevents the affected managers, including the SRO and AA, from performing their
assigned responsibilities. Other consequences could evolve; e.g., the Agency's failure to obtain
OMB clearance of Agency sponsored surveys violates the Paperwork Reduction Act and could
weaken OMB support for the Agency's mission and budget.
2. Decision Memos Lack Key Elements
Findings
The review team found that in many cases the decision memoranda for CAs and lAGs
29
-------
were lacking detail to fully support all aspects of the decision process. GAD requires a decision
memorandum to document the decision to award a CA or an IAG. The key elements of such
decision memoranda are prescribed by GAD and augmented in Chapter 4 of the ORD Policies
and Procedures manual. It should be noted that some of findings in this section of the report
relate to issues discussed in later sections where more detail is provided.
A. Discussion of Principal Purpose for Cooperative Agreements
o The discussion of principal purpose in the decision memorandum for CA CR 824308
with the University of California was weak. The review team found this to be especially
problematic in that 1) there was only one respondent to this competitive RFA and 2) Acurex was
proposed as a major subcontractor on the initial proposal, when in fact Acurex had already
advanced the design of the Hynol plant under a previous contract with APPCD. Given Acurex's
prior involvement on the project as a contractor, the review team felt it was imperative that the
decision memorandum fully support the decision for funding the project as assistance vs.
acquisition.
o The decision memoranda for CAs CR824152, CR822870 and CR822871 (all with
RTI) contain boiler-plate language to support the decision to fund assistance with the language
certifying assistance is appropriate without explaining why. The ORD Policies and Procedures
manual at 4.4-11 calls for the decision memorandum to include an explanation of why the
principal purpose is public support or stimulation. To meet that criterion the documentation of
the decision should provide details supporting the decision rather than simply certifying the
decision is appropriate.
o The decision memorandum for CA CR822641 with Harvard University did not provide
sufficient documentation to support the use of a cooperative agreement as the appropriate
vehicle. Overall, there are statements in the decision memorandum which would cause alarm
currently. For example, on page 2 it states "This research will contribute to the development of
the greatly needed methods and guidelines..." and on page 8 it reads "...provide EPA with a
consistent method for establishing such guidelines". The review team believes the correct
vehicle was selected but the wording in the initial decision memorandum could lead to another
conclusion. A later decision memorandum supporting an extension to the period of performance
was worded more appropriately indicating that APPCD is more cognizant of the rules for
differentiating between assistance and acquisition, and the need to clearly document the resulting
decision. '
B. Discussion of Substantial Involvement for Cooperative Agreements
o There was no explanation of how the Government contemplated substantial Federal
involvement to justify the use of a cooperative agreement vs. a grant in the decision memoranda
for CAs CR822871, CR823052, and CR824152 (all with RTI), CR819641 with Aarhus
University, CR824308 with the University of California, or CR822780 with Southern Research
Institute.
30
-------
o CA CR822870 with RTI states there will be close cooperation, but provides no detail
on the extent and nature of such cooperation.
* *
o The review team found during the interview process that certain POs did not
demonstrate a clear understanding of the principles of selection of instrument. For example, one
PO noted that the assistance agreement had to be a CA since the laboratory did not award grants
and another PO noted that the assistance agreement had to be a CA because EPA "didn't know
what we wanted".
C. Discussion of Peer Reviews and Reconciliation of Contrary Reviews
o CA CR823052 with RTI included a summary section on peer review which identified
the members of the peer review panel and their backgrounds while noting that there were "no
overwhelming contrary reviews". The actual peer review documentation did have negative
comments.
o The peer reviews leading to the award of CA CR819641 with Aarhus University noted
that while there was insufficient data to1 evaluate the technical merit of the proposal, the peer
reviewers based their support of the award On the reputation and publication history of the
Principle Investigator. The decision memorandum noted the lack of negative notes, but there
was no documentation of positive support explaining why the award should be made.
o The section on peer review for CA CR822870 with RTI simply stated that the external
and internal peer reviews were favorable and recommended funding. There was no discussion of
contrary reviews. (Note: Only one of the 3 peer reviewers had any substantive comments.
During the interview the PO said these comments were relayed to the cooperator after award for
consideration. There is no file documentation on this.)
D. Discussion of Apparent or Duplicative Effort
o The section of the decision memorandum titled Explanation of any Apparent
Duplicative or Excessive Effort gives the PO the opportunity to acknowledge like topics or
parallel participation. For example, CA CR823052 with RTI was for "Life Cycle Assessment to
Compare Municipal Solid Waste Management Options". That same language was used in the
statement of work for lAGs RW89936334 with DOE and RW64937353 with the Tennessee
Valley Authority (TVA), but none of the decision memoranda acknowledged or described the
relationships. While there was a significant amount of file documentation to explain the
relationship, it would have been helpful to clarify the relationships through some discussion in
the decision memorandum.
E. Identification of Statutory Authority and ORD Issue
o The decision memoranda for CAs CR822870 and CR824152, both with RTI, and
CR824308 with the University of California did not identify the statutory authority or the ORD
issue.
31
-------
o IAG DW96936393 with COE includes a decision memorandum that cites both the
Economy Act and CERCLA. The authority to enter into an LAG should be either the Economy
Act or a cooperating authority, not both.
o There is a CA with the Georgia Tech Research Institute under LAG RW69937476 with
the U.S. Department of Transportation/Federal Highway Administration; however, the decision
memorandum is silent on the statutory authorities of both parties for awarding assistance. It also
fails to include the required statement that any funds used on the CA would not be used for the
direct benefit or use of the Federal Government.
'• •' >
o The decision memorandum for LAG DW89937403 with DOE/Sandia fails to cite the
cooperation authority, nor does it identify Sandia as a contractor. During the interview the PO
stated he was not familiar with the various cooperative authorities related to LAGs. At the time
of the interview, the PO had not yet taken the ORD training for assistance and LAGs, though he
indicated he had received training from GAD.
F. Discussion of Quality Assurance Requirements
o In the discussion of QA for LAG DW96936393 with COE the decision memorandum
notes QA is addressed in the proposal and that QA plans will be developed for specific tasks. It
also states the QA section was reviewed and found satisfactory, but it does not say by whom.
The review team could find no evidence of QAO review of the award in the PO's file, but
documentation demonstrating the QAO's review was provided to the review team at a later date,
G. Discussion of Competition
' (. • •
o The review team noted the file for CA CR824152 with RTI contained no evidence that
the RFA was ever published in the CBD or Federal Register. Subsequently the APPCD EMS
provided the review team with copies of request letters from two different sources which indicate
the RFA was publicized in the CBD. However, the decision memorandum does not provide any
discussion of the CBD publication which would have established the full extent of competition.
H. Discussion of Travel
''- \..' •
o The file for LAG DW9693693 with COE included no discussion of travel though $50K
is proposed for travel. f
A fully developed decision memorandum ensures the Decision Official is aware of all the
key aspects of the proposed award. It also serves to document the Agency's decision-making
process for those who may question the decision after it is rendered. The review team
understands that many of the decision memoranda that were reviewed were written prior to
training and/or development of guidance on the preparation of such memoranda. Accordingly
the shortcomings noted here are intended to help the division improve future decision
memoranda.
32
-------
3. Records Management
Findings:
w~~~ < •
The review team found that some of the files associated with extramural agreements were
comprehensive and well organized while others were not. The files of POs and WAMs
constitute official Agency records and as such should be maintained in an orderly fashion with
full documentation of activities both prior to and after award. The ORD Work Assignment
Manager Course For Contract Management and the ORD Project Officers Course for Assistance
Management (Grants and Cooperative Agreements) each include a separate chapter defining the
POs' and WAMs' responsibilities for records management. Guidance is also provided relative to
the documentation of monitoring activities which include annotated progress reports, site visit
reports, minutes of meetings, summary of phone calls, etc. Some examples of, the review team's
findings relative to records management within APPCD follow:
i
A. Examples of Good Records Management Practices
o The PO's files for lAGs DW49936487 and DW96936393 were well ordered. The latter
was noteworthy for the inclusion of documented reviews of draft documents.
o The PO's file for CA CR822871 with RTI was exemplary from the standpoint of
comprehensiveness and orderliness. Clear evidence of PO monitoring and collaboration was
included in the file, along with an abundance of correspondence addressing issues that arose and
were addressed. >
o The PO's file for CA CR822025 with RTI fully documented the high level of
monitoring provided by the PO, including annotated progress reports, site visit reports, summary
of meetings at the laboratory, etc.
o CA CR822007 with Georgia Tech Research Corporation provided an example of
complete pre-award file documentation, including a copy of the synopsis, bidder's mailing list,
and list of those requesting the package. Monitoring was well documented in a separate
notebook, including copies of emails, marginal notes in progress reports, documentation of
technical direction, etc.
/
o The PO's file for IAG DW 57937570 with US Air Force (USAF) included
documentation of monitoring in the form of marginal notes on progress reports, memoranda for
the record on conference calls, and copies of faxes.
o The PO's file for Contract 68-D4-0100 fully documented the use of email
communication with hard copies. Marginal notes were included on the contractual document,
work plan and progress reports.
B. Examples of Records Management Practices that Could Be Improved
33
-------
o The file for CA CR822780 with Southern Research Institute did not include any
monitoring documentation. There were multiple copies of the two annual reports but there was
no documentation of review by the PO. During the interview the PO indicated that he received
and marked up draft copies of the annual report, but that was not evident during the file review.
There were no copies of reports recording financial progress. The terms and conditions for the
agreement require PO participation in planning the field tests, selecting the test conditions, and
participating in visits by the grantee to sites at which experimental programs are in progress.
There were no records of such actions. The PO indicated that sufficient funds for such travel
have not been available for the past three years. There is also a term and condition that the EPA
PO will assist in the overall operation, provide review comments, monitor the project for
timeliness, and budget. The only monitoring notes found were on the PO's desk calendar. Such
notes should be transcribed into the official CA file.
o The file for Contract 68-D1-0008 with International Fuel Cells was disorganized and
could have benefitted from a file content checklist. Some key documentation that was missing
includes:
— Evidence of competition.
— Support for the extension of the contract for 2 years resulting in a seven year period of
performance.
— No independent government cost estimate developed prior to the solicitation.
— Though the PO indicated during the interview that he holds infrequent telephone
meetings with the contractor and reviews reports, there was no file documentation to confirm.
o The file for CA CR822641 with Harvard University contained two faxed copies of
progress reports, but no other evidence of PO monitoring. During the interview, the PO
indicated he has had trouble obtaining progress reports due to a low funding level and
accomplishes monitoring of the agreement through phone calls and infrequent travel; however
there is no evidence of this in the file.
o During the interview the PO for IAG DW89937403 with DOE/Sandia indicated
progress is monitored through conference calls and review of invoices, though documentation of
those activities is lacking in the file. The PO stated the two agencies jointly review and evaluate
data, but there is nothing in the file to support that statement. Progress cannot be monitored
through the review of progress reports since the agreement did not call for them.
o The PO's files on CA CR822870 with RTI lack evidence of monitoring by the PO.
The CA calls for work plans to be submitted after award for specific tasks, but none were
available in the file. The agreement does not call for submission of progress reports.
Poorly organized and/or incomplete files demonstrate a lack of compliance on the part of
APPCD with records management responsibilities and can inhibit the effective management of
the Division's extramural agreements. This is especially critical when personnel responsible for
managing agreements are replaced or when the files are reviewed by outside personnel such as
auditors. Clearly some of the Divisional personnel are aware of their records management
responsibilities and take them seriously.
34
-------
4. Divisional Personnel Demonstrate Effective Monitoring/Addressing Problems
The review team identified several instances of a high level of monitoring activity that
enabled the Division to address problems early on. Effective monitoring of extramural
agreements is stressed as a key responsibility in both the ORD Work Assignment Manager
Course for Contract Management and in the ORD Project Officers Course for Assistance
Management (Grants and CooperativeAgreements). Examples include:
o Contract 68-D4-0005 with Acurex has a number of indicators of excellent contract
management practices.
r
— During the interview the PO indicated she reviews the contract practically every day,
and relies heavily upon experience when reviewing contract and work assignment issues.
— The file documented the resolution of an instance of improper data manipulation by an
Acurex employee. While it was Acurex that officially identified the problem, in fact NRMRL
QA reviewers identified the existence of a data problem months earlier. The files show prompt
action by both parties, NRMRL and Acurex, to bring the problem to light and to deal with it
promptly and effectively.
— The PO and WAM files show careful attention to requiring, reviewing and approving
Work/QA plans as an integral part of startup of the assignments. There was evidence of
substantive review and in some cases of negotiation and revision when appropriate.
— The work assignment on continuous emission monitoring (2-52) showed WAM
awareness of the need for good documentation and approval of guest workers. The WAM was
careful to secure appropriate clearances and involved the Human Resources Division
appropriately.
— The files indicate consistent use of independent government cost estimates. During
the interview she indicated her intent, to put together a spreadsheet in the near future. (The
review team noted that the PO recommends use of a single overall loaded labor rate. In
reviewing the PO files, we found that there is a very large range of actual incurred cost per hour
on the various work assignments, based in part on a variety of labor mixes and ODG. The range
is from $30/hour to over $100/hour. The use of a single loaded labor rate makes it difficult to
accomodate such wide variances which results in significant differences between the
Government's and the contractor's estimates for individual work assignments. While the PO's
IGCE format does allow for special variations, it would be worthwhile to evaluate future use of a
more detailed estimating process.)
-- The PO and WAM files that were reviewed were very well organized and
demonstrated active management and receipt of required outputs. The files for WA 1-42 and 2-
30 were particularly well organized and easy to review. The PO documents the review of
deliverables with memoranda to the contractor (cc: CO) and marginal notes on progress reports.
35
-------
-- The files clearly demonstrate careful administrative review of invoices by the PO, and
investigation or suspension when appropriate. Understandably, during the furloughs of the past
year, there were a few occasions when invoices were not reviewed and acted upon within the
time allotted by Financal Management Division. However timing was good once the furloughs
ended and, as noted above, the reviews were meticulous. Since the timing of such reviews is an
SRO performance standard, the speed and quality of the PO's review deserves notice.
- The files showed the PO works extensively with the EMS and the CO to deal with
contract problems. There was good cooperation in such matters as issuance of Work
Assignments and administrative contract amendments, and in resolution of problems such as the
data manipulation matter described above. The PO also meets with the contractor every two
weeks to address contract issues. The PO was careful to request the formal appointment of an
alternate PO during her absence on sick leave.
o The PO for CA CR822871 with RTI was able to demonstrate through the file review
and interview that she has been and continues to be on top of the issues related to her CA. One
example is a problem that arose relative to the availability of SERDP funds that were already
obligated to the agreement. The provider of the funds threatened to pull them back if not spent in
a particular time period, but the PO researched the issue and convinced them it was not required
nor in anyone's interest to do so. She also has had to deal with numerous issues associated with
clearance procedures to conduct a site demonstration project on a military base. Her prompt
attention to these requirements has significantly reduced the potential for delay in completing the
research project.
o The review team found the management of the Delivery Order under the FAIR contract
(68-W5-0065) to be exemplary. The DOPO was cleatrly concerned with the appearance as well
as any actual violations of regulation, and she has instituted procedures to provide arms-length
dealings between Government and contractor employees. She displays a strong working
knowledge of contract rules and regulations tempered by a concern that there may be more
information that would allow her to do her job better. She utilizes computer resources to the i
maximum extent for day-to-day operations such as tracking the issuance, status and completion
of work orders. This system is set up on the local area network with access by the contractor to
indicate when jobs are completed. Thus contract management data is available in computer
format which reduces unnecessary paper or time-consuming human interaction. She appears to
seek a balance between protecting the Government's interest through monitoring and allowing
the contractor to perform as provided under the contract.
5. Lack of Financial Detail Inhibits Financial Monitoring of CAs and lAGs
Findings;
The review team noted that for the majority of CAs and lAGs reviewed, the level of
financial detail provided by the recipient (under CAs) or other Agency (under lAGs) was
insufficient to enable the PO to fulfill his/her responsibilities relative to monitoring the financial
aspects of the agreements. In many cases the PO is apprised only of the amount of funding that
36
-------
was spent and what remains unspent. Likewise there was little documentation in the file to
demonstrate that APPCD provides other agencies with sufficient spending information on funds-
in LAGs.
The Office of Inspector General in its Audit Report No. E1FMG4-13-0061-5400051
dated March 31, 1995 found that POs were not obtaining detailed cost information under funds-
out LAGs even though a detailed breakdown of the incurred costs is required when other
Agencies request payment. The PO's responsibilites relative to financial management are set
forth on page 4.v-l 1 of the LAG Compendium and include both assuring the receipt of the other
Agency's detailed cost information and reviewing that cost information to ensure that project
costs, as listed within the budget category and limits specified in the LAG, are acceptable. Under
LAGs, the PO is required to approve payments to the other Agency. In its report the OIG noted
that if such detailed information is not provided, POs are required to notify the Cincinnati
Financial Management Center (CFMC) to suspend payment. As noted, the review team found
numerous instances where detailed cost information was not provided by the other Agency; but
there was no evidence that the PO requested CFMC to suspend payment. The OIG report
concludes: "Without detailed invoices, project officers will be unable to comply with the FAR
prohibition against paying fees in excess of costs." ,
Unlike LAGs, POs under assistance agreements are generally not called upon to approve
the payment requests from the recipients. Furthermore the amount of information required to
support amounts claimed on recipient payment requests is limited by OMB Circulars A-102 and
A-l 10. However, POs are required to review Financial Status Reports if required by the
agreement to assess the progress against the expenditure of funds. Furthermore it is currently
recommended within ORD (through the assistance training course) that the terms and conditions
included in assistance agreements clearly call for detailed cost information in the periodic
progress reports. Such information would provide the PO with the opportunity to monitor for
improper costs being claimed; e.g., costs claimed for the acquisition of equipment that had not
been authorized. It also provides the PO with the means to measure technical progress more
effectively by understanding not just how much was spent, but for what purposes.
The review team recognizes that the Agency needs to improve its financial monitoring
under assistance and interagency agreements. The review team also recognizes that the
agreements reviewed were awarded prior to the promulgation of new policy within both the
Agency and ORD. resulting from criticism rendered in recent OIG audit reports, and that training
on financial monitoring was lacking until FY96. However, we concur that financial monitoring
is a key responsibility for POs under both assistance and interagenqy agreements.
Recommendations;
The review team makes the following recommendations:
4.2.1 The Director, APPCD should take action to ensure all of the Division's open extramural
award and management files reflect all required approvals. Where evidence of approval is
missing, we recommend that the Director determine whether the approvals were obtained, and
37
-------
obtain documentation if possible. If there are instances where required approvals were not
obtained, appropriate action needs to be determined on a case-by-case basis with input from the
EMS.
t t
4.2.2 The Director, APPCD should take appropriate action to ensure that all approvals on future
actions are obtained, that the necessary documentation is retained, that files are well organized,
and that all files include the required and expected evidence of monitoring; i.e., records of phone
calls, letters, and reports with documentation to show review by the PO. File plans and/or
checklists could be one approach to use. For example, handouts 5-8 and 11-3 of the ORD
Project Officers Course for Assistance Agreements provide examples of checklists which have
been used successfully by other organizational units in ORD.
; f
4.2.3 The Director, APPCD should take appropriate measures to strengthen the Division's
decision memoranda for both CAs and LAGs. Other organizational units in ORD have
experienced success by providing access to a template decision memorandum maintained by the
EMS on a shared drive. However the review team cautions against being overly reliant on the
boilerplate contained in a template decision memorandum. The details provided within the
decision memorandum need to be drafted to fit the specific circumstances related to the proposed
action.
(Note that the Director, GAD issued a memorandum with an effective date of 10/1/96 that
redefined the requirements of the decision memoranda for LAGs. The template for IAG decision
memoranda would address those requirements.)
4,2.4 The Director, APPCD should take appropriate action to ensure that all future assistance
and interagency agreements include a term and condition requiring detailed financial information
to be submitted on a periodic basis; e.g., quarterly. In cooperative agreements, where POs are
seldom called upon to review payment requests, financial details included in progress reports
should be sufficient for monitoring purposes. For LAGs, the detailed financial information
should coincide with the other Agency's request for payment so as to provide the PO with
sufficient information to effectively review the payment request. The level of detail should
conform to the categories used to establish the agreement which generally include direct labor,
fringe benefits, travel, equipment, supplies, indirect costs, etc. Note that if subcontractors are
included as a significant partner in the agreement, then detailed financial information from the
subcontractors should also be required,
4.2.5 The Director, APPCD should take appropriate action to ensure that LAG POs are aware of
their responsibility to request the suspension of payment when detailed cost information is not
provided. The review team also recommends that the financial responsibilities of POs under
assistance agreements be reemphasized and that the EMS provide assistance to those POs who
believe that more detailed information is needed for them to effectively monitor the financial
aspects of their current agreements which may lack language calling for such information.
DIVISION GENERAL RESPONSE TO CROSS CUTTING FINDINGS: We note that
much of the ORD guidance for file documentation and Decision Memorandum content came into
38
-------
being after the award of the actions reviewed. For example, most of the actions reviewed were
awarded before the issuance of the July, 1994, version of Chapter 12 of the ORB Policy and
Procedures Manual. We also note that a number of the items discussed above under "decision
memos..." are still not covered by the 1995 version of Chapter 4 of that Manual (e.g., there is no
guidance regarding discussion of substantial involvement, competition and travel). In any event,
please be assured that we plan to improve the quality of our documentation and files as
recommended and we propose the following actions in response to the recommendations:
RECOMMENDATIONS
, - ./ •
4.2.1 DIVISION RESPONSE: The Review Team recommended that we ensure all open
extramural files reflect all required approvals.
ACTION TAKEN: We are instituting a program to review all of our active extramural files
and we will obtain any missing documents as time permits.
4.2.2 DIVISION RESPONSE: The Review Team recommended that we ensure approvals are
obtained and documentation is retained for all future actions...all files are well organized and
include evidence of monitoring.
ACTION TAKEN: We are preparing file checklists for various types of actions (competitive ,
and non-competitive cooperative agreements, reimbursement and disbursement lAG's) and, after
coordinating with NRMRL, we will distribute these and provide assistance to the APPCD Project
Officers in utilizing them. We believe that this, along with a diligent review and documentation
of the approval process for future actions, will solve much of this perceived problem.
.1 .
4.2.3 DIVISION RESPONSE: The Review Team recommended that we strengthen decision
memos.
i ' ' r
ACTION TAKEN: In March of 1996, we issued guidance and a standard format (see
Attachment 4) for decision memos for both cooperative agreements and LAGs based on the
requirements of Chapter 4 of the ORD Policy and Procedures Manual and the text for the ORD
Project Officer's Assistance Course. We believe that this, along with a diligent review and
documentation of the approval process will adequately deal with this issue.
4.2.4 DIVISION RESPONSE: The Review Team recommended that we ensure future coops
and lAGs require detailed financial reporting.
*
ACTION TAKEN: In the past, the financial reporting requirements have been specified by
GAD. However, in December of 1996, we issued guidance regarding terms and conditions for
cooperative agreements, including examples of financial and technical status report
requirements (see Attachment 5). In addition, we will attempt, during the above mentioned file
reviews, to identify needs and to initiate amendments where appropriate to obtain more detailed
financial reporting, within the limits set by 40 CFR 31.40,40 CFR 160, and OMB Circular A-
110 Part 50.
39
-------
\
4.3 COOPERATIVE AGREEMENTS
INTRODUCTION. The-review team reviewed seventeen current APPCD CA files and
interviewed seven POs. The review covered the following CAs (with those POs who were
interviewed marked with an asterisk):
CANO. COOPERATOR PROJECT OFFICER
CR823020 Georgia Tech Research Corp Carl Ripberger
CR822641 Harvard University John Chang* x
CR824049 Midwest Research Institute Robert McCrillis
CR822007 Georgia Tech Research Corp Kelly Leovic
CR822025 Research Triangle Institute Kelly Leovic
CR824152 Research Triangle Institute Chester Vogel*
CR823632 University of Minnesota Russell Kulp
CR824308 U. of California-Santa Cruz Robert Borgwardt*
CR824639 Tech.Institute for Manufacturing Electronics Charles Darvin
CR824257 Research Triangle Institute Carlos Nunez*
CR822227 Texas A&M University Robert Hendricks
CR822780 Southern Research Institute Samuel Rakes*
CR823052 Research Triangle Institute Susan Thorneloe-Howard
CR819641 University of Aarhus Mark Mason
CR822870 Research Triangle Institute Leslie Sparks*
CR822871 Research Triangle Institute Carol Purvis*
CR824302 North Carolina State University JohnWasser
The specific findings and recommendations follow.
1. Scopes of CAs Modified Without ORD and/or GAD Approval
Findings; .
t j'
The review team detected instances where CAs appeared to have been modified during
performance without being submitted to designated ORD officials for approval and/or being
transmitted to GAD for processing as a formal amendment to the agreement. The ORD Policies
and Procedures manualprovides guidance for approval of changes outside the original scope of a
previously approved assistance agreement. In conjunction with ORD's guidance, both the GAD
and ORD training manuals fdr POs provide additional instructions along with illustrations on
changes that require issuance of a formal amendment by GAD. The ORD guidance and
delegations of authority are established at Chapter 4.4-15, post award phase, item (b) which reads
as follows:
(b) Supplemental funding with additional work and/or years outside the scope and outside
40
-------
the original approved project purpose yielding a change in scope and funding level
requires ORD approval as follows:
,, May Be
Function Concurrence Approvals Redelegated To
a) <$1 million LD/CD/OD Deputy, DD
b) $ 1 million to $5 million SRO LD/CD/OD Deputy
c)> $5 million SRO and AA/ORD None
LD/CD/OD
Furthermore, 40 CFR 30 (in both the former and recently revised versions) requires
approval of the EPA award official for:
— Change in the scope or the objective of the project or program (even if there is no
associated budget revision requiring prior written approval).
-- The need for additional Federal funding.
Examples where changes had occurred on specific agreements without evidence of prior
approval being obtained from the appropriate ORD official or issuance of a formal amendment
from GAD are outlined below:
o The review team identified a number of instances under CA CR822870 with RTI
where the agreement should have been, but was not, amended by GAD to reflect changes in the
scope. This CA identified numerous independent research projects (minimum 13), most of
which were not initiated due to budget constraints. The projects were often described in general
terms indicating some preliminary research would have to be conducted before a project
narrative was finalized. During the interview the PO identified four of the projects that were
initiated and indicated two of those were scaled back. He also indicated a new project was
added. The new project could only remotely be tied to the area of the proposal that was
identified. The specific findings related to this CA are detailed in Exhibit 1. The review team
believes that the agreement required a formal amendment to 1) reduce the number of projects
associated with the agreement, 2) to scale back the scope of projects that had been approved and
3) to initiate projects that were hot defined in the original research proposal.
o The review team found similar problems'under CA CR824152 with RTI. The original
scope of this CA was established to conduct evaluations to assess the performances economics,
and reduced emissions potential 6f newly developed low-emitting adhesive formulations. Under
the initial stages, these product evaluations were to concentrate on gathering data on lower-
emitting adhesives as replacements for conventional ones within two industry sectors:
automotive assembly and furniture manufacturing. This data was then to be provided to EPA
and the technical advisory group to assist with research scoping and to be used to narrow the
focus to specific applications within the two previously mentioned industry sectors. Following
discussions between EPA and RTI to narrow the focus to specific applications, RTI was to
conduct bench scale tests on new or emerging low-VOC/low-HAP adhesive products. The
scope of this CA was changed approximately one year after award without LD approval or the
issuance of a formal amendment through GAD. Documentation within the file indicates that
41
-------
another ORD organizational unit (NCERQA) provided the funding, commitment notice, and
proposed research scope to initiate this post-award change. This change was primarily a
continuation of work initiated under a previous CA with RTI (CR818419) which expired on
9/30/95. The recipient was instructed to 1) continue it's development and expansion of EPA's
Enviro$en$e (ES) information system, 2) continue support of Environ$sen$e World Wide Web
activities, 3) continue daily and periodic maintenance of the Web server containing SAGE and
the Navy P2 Library, 4) update and add other data bases to the Solvent Alternative Umbrella, 5)
participate in the development and implementation of a DOD ES deployment plan which
includes the development of multimedia training/deployment at various Navy Environmental
Leadership Program (NELP) workshops, 6) participate as a member of the Navy Facilities
Engineering Service Center, DOD and the ES working group, and 7) assist with the development
of various models, videos and other training tools to be used in the NELP workshops. The
review team finds that these tasks clearly fall outside the original scope as identified in the
approved research narrative. Furthermore, the file documentation indicates the new work was
performed at the direction of and primarily for the direct benefit of the Agency and as such was
more appropriate for acquisition than assistance. The review team also has concerns about the
competitive process since it appears that NCERQA utilized this existing CA to circumvent
competition. >
o The review team found that CA CR824308 with the University of California-Riverside
was downscaled without a formal amendment. The complete project as conceptually planned
and presented in the proposal was to include 3.5 years of fabrication, installation and testing at an
estimated project cost of $6.5 million. However, documentation contained within the file
indicates that funding for this project was a problem from the very beginning. Initially EPA's
available funding for this project was only $743,000 with the intention of securing funds from
other sources that were not specifically identified. To date EPA has provided a total of $1.3
million to this agreement for the first two years of the project. There are several memoranda in
the file from the PO to the recipient indicating the lack of funding and emphasizing the
importance of completing the gasifier project before undertaking any other work. This
agreement should have been formally amended to reflect this apparent lack of funding and
resulting downscaling of the project. v
Failure to secure appropriate ORD approvals to proceed with significant revisions to
previously approved assistance agreements may result in situations where EPA is funding (or not
funding, as the case my be) those projects that ORD management believes are of the greatest
benefit to the public at large. Furthermore, review of changes to the scope of CAs by the EMS
would reduce the chance that such changes create vulnerabilities for the Division. Finally,
failure to secure GAD approval of such changes though a formal amendment creates a situation
where the Division is in violation of the CFR.
*
2. Assistance Agreements Improperly Used to Acquire Direct Agency Support
Findings.
The review team found a number of instances wherein assistance agreements appear to
42
-------
have been used to acquire direct support for the Agency contrary to the provisions of the Federal
Grants and Cooperative Agreement Act (FGCAA). EPA Order 5700.1 expands upon the
FGCAA by providing guidance for distinguishing between assistance and acquisition. Section
6.b(l) of the Order includes the following guidance:
"Direct Benefit or Use - In applying the principal purpose test, offices and laboratories
must determine whether the government is the direct beneficiary or user of the activity. If
EPA provides the specifications for the project; or is having the project completed based
on its own identified needs; or will directly use the report or result of the project (for
example, by incorporating or relying on information developed by the project in "writing
EPA guidance or standards) then, in most cases, the principal purpose is to acquire
property or services for the direct benefit or use of EPA and thus, a contract relationship
exists."
Examples of instances where the review team found evidence of improper use of assistance
agreements include:
o The review team found that CA CR822870 with RTI appears to have been used, at
least in part, to acquire direct support for the Agency based upon the following:
— The need for this specific research project was identified by an Agency program office.
— The Agency provided the specifications for this project.
— Deliverables include art EPA guidance document and an EPA report.
Exhibit 1 to this report provides full details of the findings to support this conclusion.
i
o Another assistance vehicle that appears to have been utilized to acquire direct support
for the Agency was CA CR824257, again with RTI. The review team found that reports and
correspondence contain considerable language suggesting that the purpose of the agreement is at
least significantly for the direct benefit of EPA. Examples include:
i
- 11/6/95 RTI Report "EPA would like RTI to look at the former OCB's strategic plan
(1994) to critique it and possibly identify additional work needed to improve it."
— (Same report) "[EPA] is interested in a comprehensive and integrated research
program in paint and coatings, but what exactly this means needs further definition."
- (Same report) "[EPA] wants to get together...to start to map this [fundamental
research projects] out and get some work going."
-- RTI Report 7/16/96 "...This proposed work [completion of the "CAGE" document
that was begun 24 months earlier under another agreement] would accomplish the
original goal set forth by EPA in developing the CAGE."
- 4/17/96 RTI to EPA letter "[EPA] requested that I prepare an estimate to do a
(short) study of the stability of styrene in (heated) Tedlar bags."
-'- RTI report 10/96 "RTI has been asked to write a paper and present CAGE at the
upcoming A&WMA...conference....Originally the Project Officer was going to write the
paper and perform the presentation. The abstract was written and submitted by EPA-"
43
-------
- Rtl Report 9/96 "In anticipation of regulatory development efforts for the textile
industry, it is important that pollution prevention be considered now so that research can
be completed before and includes (sic) as an integral part of the proposed rule."
(emphasis added).
RTI Report 7/96 "The purpose of this research area is to complete various final research
reports that were initiated during a previous cooperative agreement." And later in a • v-
different area of the same report: "The purpose of this research area is to complete the
final research report for the innovative ink-feed systems project that was initiated under a
previous cooperative agreement...."
These statements within the CA file could lead to the conclusion that a significant amount of
effort put forth by RTI was for the direct benefit of the Agency and as such should not have been
performed under a CA.
o CA CR822227 with Texas A&M was established to study the performance of
refrigerants for replacement of HCFC-22. The EPA peer reviewer for this CA stated "successful
completion will enable EPA to successfully regulate phaseout of HCFC-22", implying a direct,
immediate need in support of Agency regulatory activities.
o CA CR824152 with RTI, as discussed in the previous section, had indications that the
work was performed at the direction and for the direct benefit of the Agency.
The use of assistance agreements to acquire direct support for the Agency is contrary to
the requirements of the FGCAA, as well as other Federal and Agency regulations, policies and
procedures. This type of activity undermines ORD's credibility with Congress and the OIG and
jeopardizes our continued efforts of internal reform relative to management vulnerabilities.
3; Documentation of Competition Under CAs Could Be Improved
' . ' S '
./ ' , . . •
Findings. '\ f-
At least five competitive CA files were included in the review which did not contain
adequate documentation of the competition. The Assistance Administration Manual (EPA Order
5700) discusses at page 15-2 the requirements for documenting the decision process including
the need to "...establish a clear and complete record of each decision process and retain this
documentation in each official file." Not too much should be read into the fact that each of the
examples cited below was awarded to RTI, since all five had different POs. Also, the files
selected were weighted somewhat on RTI because of the initial observation that RTI had a great
deal of actions with the laboratory. Specific findings follow:
, «
o CA CR822870 was established to study various building systems for potential methods
of reducing indoor air pollution. The decision memorandum stated that the award resulted from
a competition,, but that was all. Neither in the decision memorandum, nor elsewhere in the award
file was there any discussion of how many or who proposed, or of what comments were made by
the peers, or of why RTI was selected. It is possible that there was a separate file that was not
44
-------
provided for review, but the decision memorandum should contain or refer to a summary of the
process.
i '
o CA CR822871 was established to perform a demonstration on energy conversion from
biomass. While the file did contain the peer review score sheets and a chart of the scores, the
decision memorandum did not contain a discussion of the reviews nor of the selection decision.
While the file would.permit the reviewer to conclude that the selection of RTI was appropriate,
there is no clear record of the approval official's decision rationale.
o CA CR822025 was established to apply pollution prevention techniques to indoor air
pollutant emissions from office equipment and other office products. In areas other than
selection, this file was well documented. While there was a chart of the peer scores that showed
RTI as the highest rated applicant, there was no evidence of the substance of the reviews, and the
application from RTI did not contain a budget. Also, while the RFA and SOW use the term
"research" from time to time, a layman's review of the application and SOW do not indicate that
any particularly substantive research was in fact proposed and available for peer review. It is
essentially a proposal to do data searches and conferences.
The common problem in all of these was the absence of clear documentation in the file
regarding the competition and the rationale for selection.
o CA CR824257 was established to study the potential for reduction of VOC/HAP
emissions from various coating processes (paint, etc.). The decision memorandum does indicate
that four applications were received, and from whom, but there is nothing in the decision
memorandum or elsewhere in the file to demonstrate the evaluation and selection process. From
the file review, it was difficult to follow the actual steps in the process - dates of release of RFA;
close date for proposals, date of submission to peer review, and of the peer report. Discussions
with the EMS indicate there were two RFAs, an initial mailing in April 1993 that indicated
funding in the amount of $1.2M, and then another issuance in November 1993 which indicated a
higher level of funding at $4.5M. The EMS explained that there was only one response to the
initial issuance which was not considered satisfactory, and subsequent to the receipt of that
application more funding became available. For that reason the second solicitation was issued.
There is little if anything in the file other than the two RFAs to explain the rationale for
cancelling the first solicitation and issuing the second at a higher funding level. Additionally,
there are no copies of the peer reviews on the three who were not selected, nor any discussion of
their findings. There is no rationale for why RTI was considered to be superior to the others.
There also is no evaluation of the budget proposed by RTI.
There is also a concern about the research focus in the RFA versus the research narrative
included in the agreement. The RFA called for proposals for projects to improve existing coating
methods and to develop new ones. But the RTI application contained little in the way of specific
research proposals, and instead proposed to survey coating industry representatives to find out
where the future research needs lie, and only then to propose specific research. It is not clear
whether the non-selected applicants proposed specific research or whether they also understood
that specifics could be determined later. Nor is it clear that all proposals therefore were reviewed
45
-------
on the same basis. In addition, the RTI proposal indicates at one point that RTI and EPA had
already jointly sponsored a focus group meeting to determine research needs. Some comments
in the proposal, and more in the post award correspondence indicate that much of the RTI effort
is directed toward the completion of reports on work that was sponsored under other agreements.
These findings are not necessarily inappropriate, especially on an individual basis. But
when taken as a whole, the file does not present a clear picture of a fair competition:
-- an RFA whose value triples without a documented explanation;
— and whose scope emphasis changes without explanation;
- where it appears RTI and EPA held discussions under a predecessor CA which led to
the scoping of the research needs for the new CA;
- where the winning application is not particularly specific on research to be done;
- where the file does not contain information on the specifics of the applications of the
other competing applicants;
- and the peer reviews are not based on an approved plan, or lack detail, or are not
clearly unbiased.
o CA CR823052 for studying municipal waste management options, had more than an
absence of documentation. The facts available in the file suggest that there were substantive
problems with the conduct of the peer review and with the selection process itself. RTI was not
the first choice for 4 of the 5 reviewers. RTI was voted highest by only one, and two other
applicants received top rating by two reviewers each. There were questions raised about RTI
during the peer review that were not addressed in the file; for example:
'f~
"...the description of the MSW stages does not reflect accurate understanding of the LCA
concept and how it should be applied to the MSW model."
"...also have significant concern regarding statements about supporting graduate students
and research at NCSU and University of Wisconsin. These have not been identified and
one can be assured that both NCSU and particularly University of Wisconsin will find
reasons to support graduate students. This is not what this study is about! I have
concerns that there is not sufficient representation by local governments and to [sic] much
by very vocal groups against combustion, landfilling and any other method that is not
some reduction/recycling. ...I do not believe that their main team... is still very strong.
Many of my comments on the preproposal still are valid. While they may be able to
complete the work, I do not have the confidence that it will be the best we can get for the
dollars available. Break in cost sharing gives them an unfair advantage."
One reviewer sent the PO a separate memo expressing concern over awarding to RTI since their
peer review group was "uncommitted if not hostile to waste-to-energy", which was the focus of
the agreement.
Not only do thesef comments raise concerns about the adequacy of the peer review
process (as discussed in the following section), but they leave a large gap in the credibility of the
competitive process. The decision memorandum simply addressed the membership of the peer
n ,
46
-------
panel and stated that there were "no overwhelming contrary views."
Subsequent to the on-site review, the APPCD EMS explained that there was controversy
on the peer review panel with 2 of 5 reviewers favoring one approach by one applicant, 2 of 5
reviewers favoring another approach by another applicant, and the fifth reviewer favoring RTI.
Since RTI was rated first by one reviewer, second by three other reviewers, and third by the fifth
reviewer; the decision was made after lengthy discussions to go with RTI as the middle ground
between the two polar po'sitions. The EMS told the review team that he had counseled the PO on
the necessity to fully document this decision, but agrees the file does not contain much to explain
the selection decision. Such documentation would have helped to alleviate concerns that the
wrong applicant had been selected for award.
The review team believes that APPCD's failure to adequately document the competitive
process leaves the Division vulnerable to criticism that the process is not fair, particularly in
consideration of the number of CAs awarded to RTI. (Approximately 25% of the active CAs
listed in OMIS on 9/23/96 were with RTI, and approximately 40% of the dollars associated with
the active CAs were with RTI.) If the-perception is established that RTI is a favored institution,
then other qualified recipients may be reluctant to compete for APPCD's assistance opportunities
which in turn would diminish the opportunity for the Division to fund the best research possible.
4. Weak Peer Reviews Minimize Value of Input
Finding
The review team found that many of the peer reviews that were conducted on CA
proposals failed to achieve the goal established under ORD Policies and Procedures, Chapter
14.2 which reads in part at 14.2-3:
"The purpose of peer review is to identify any technical errors or shortcomings or
unresolved issues in a preliminary product so that the revised work product will be based
on sound technical information and analyses. It should be noted that peer review is a
process for enhancing the scientific or technical work product."
The peer reviews that were conducted on APPCD's CA proposals often lacked substantive
narrative that could have been used to strengthen the research projects which the division chose
to fund. Specific findings include:
o CA CR822870 with RTI was awarded competitively. RTFs proposal covered 13
separate tasks or sub-tasks as defined, within the proposal (over 30 pages). Two external and one
internal peer reviews were located in the file, none of which were over 3 pages in length and
limited in specifics. Two of the peer reviews included no critique of any of the specific projects
being proposed; rather they stated in general terms that the research would be beneficial and that
both the personnel and facilities were appropriate. One of the external reviewers offered
criticism on 3 of the tasks proposed, but there was no evidence in the file that these comments
were addressed by RTIprior to award. In the decision memorandum section entitled
47
-------
Reconciliation of any contrary reviews it was stated succinctly: "There are none." However, as
noted above, one reviewer did offer specific criticism of the proposal. During the interview the
PO stated that he believed the comments from the one reviewer were provided to RTI after
award; but the review team found no file documentation that would verify that observation, nor
any providing RTI's response.
o CA CR822871 with RTI was awarded competitively. Thirteen pre-proposals were
received and peer reviewed with RTI being selected for award and subsequently requested to
submit a full proposal. Scoring sheets were used for the peer reviews based upon the pre-
established evaluation criteria. The documentation for the peer reviews in most cases lacked
substantive comments on the proposed technical approach. For some of the proposals, some of
the reviewers simply returned the sheets with scores assigned but no supporting narrative. The
PO indicated that it was difficult to solicit extensive narrative from peer reviewers who are
serving in a voluntary capacity.
o CA CR824257 with RTI was awarded competitively. Four applications were received
and RTI was selected for award. The file contained four peer reviews of the selected applicant,
but none for those who were not selected. One of the reviews had no Substantive comments;
only some unsupported scores. Two of the reviews contained some substantive comments, and
the other had a few. Overall, the reviews did not appear to involve a particularly impressive
analysis of the RTI application with little comment on the specific scientific quality of the
research to be done. Rather the reviews concentrated largely on available personnel and
facilities. As noted, there are no copies of any of the reviews of the three applications that were
rejected, so there is no way to determine if the selection of RTI was the result of a solid
evaluation.
o CA CR824049 with Midwest Research Institute was awarded competitively. The peer
review was insufficient inasmuch as it was based upon phone calls and a memo. The peer
review lacked detail and had no rating scale. The decision memorandum indicated there was
competition, but there was no evidence of same in the file so it is unclear what role the peer
review played in source selection.
o CA CR823052 with RTI was awarded competitively. Though the decision
memorandum indicated there were "no overwhelming contrary reviews", the file included
significant comments that should have been addressed as discussed in the previous section.
o The summary rating sheet provided in the file for CA CR822780 with Southern
Research Institute included multiple changes that were not explained.
o For CA CR819641 with Aarhus University, the peer reviewers noted that there was
insufficient data to evaluate the technical merit of the proposal but based their support of the
award on the reputation and publication history of the Principle Investigator. The file included
multiple rewrites of the proposal and the decision memorandum noted a lack of negative notes,
but there was no documentation of positive support.
48
-------
In addition to providing an opportunity for the Agency to strengthen the scientific merit
of the research supported through assistance, the peer review process also serves to substantiate
the source selection process within ORD's competitively awarded assistance agreements. As
discussed in the preceding section of this report, the lack of substantive peer reviews lessens the
credibility of the selection process.
A corollary concern relates to those assistance agreements wherein the recipient is
required to submit work plans for specific research projects after award. The review team found
little if any documentation indicating that these work plans are peer reviewed for scientific merit
before the recipient is allowed to proceed with the research project. The review team concluded
that the process of requiring work plans for the specific research projects after award contributed
to the lack of substantive comment from the peer reviewers on the initial award. Many of the
proposals or pre-proposals lacked specificity as to the research that was proposed because the
first step was often established as a study of the problem before deciding, in cooperation with
EPA, upon the exact nature of the research to be conducted. Once that decision was made, the
recipient then developed a specific research plan or in some cases (see Exhibit 1) the recipient
was provided with a Statement of Work that had been generated by EPA. This left the reviewers
of the application with little to review other.than the general discussion of the issue(s), the
qualifications of the proposed personnel, and the adequacy of the facilities.
The review team recognizes that the examples cited here were for awards made prior to
the establishment of ORD Policy and Procedures regarding peer review, and that the current
policies and procedures, guidelines, training manuals, etc. are lacking or inconsistent with regard
to the issue of obtaining substantive peer reviews of GA applications. The peer review policy is
currently evolving within the various ORD laboratories and is considered a high priority by QRD
management, including the Assistant Administrator. This review team is aware of other ORD
organizational units that have encountered difficulties in obtaining effective peer reviews,
particularly under competitive solicitations where 1) a large number of proposals or pre-
proposals are received, 2) where the number of peer reviewers is limited, and 3) where the peer
reviews are conducted on a voluntary basis. However there are other ORD organizations that
have procedures that result in effective peer reviews. ,
Poorly conducted peer reviews can adversely impact the validity of the selection process
which in turn could influence qualified research entitities to refrain from competing for the
Division's assistance agreements. Furthermore, the lack of substantive peer review comments
negates an opportunity for the Division to enhance the quality of the research projects it chooses
to fund.
5. Root Cause: Funding Non-specific Research Projects
Findings.
The review team believes that one of the primary causes contributing to the four issues
discussed above is the Division's practice of awarding CAs based upon a broadly worded, non-
specific research narrative which is supplemented after award by the submission of work plans
49
-------
developed collaboratively by EPA and recipient personnel. For example, the following special
condition was included in CA CR822870 with RTI:
"1. The EPA project officer and the Research Triangle Institute project manager will develop
detailed work plans and quality assurance plans for each task in the cooperative, agreement with
emphasis on cooperative investigations."
The review team found that a significant number of APPCD's CAs were awarded with
similar special conditions or requirements. While the review team is not aware of any written
policy that would prohibit the practice of allowing recipients to flesh out their original proposals
after award, the team none-the-less feels the practice raises serious questions as to the validity of
the competitive process and to the efficacy of the peer reviews. Furthermore, the review team
found that in some instances EPA defines the research that it wants accomplished for its direct
use. The review team also believes the practice of requesting post-award work plans resulted in
some instances of work being added that did not fit within the scope of the awarded agreement.
The review team has been advised that due to budgetary contraints in FY 96, NRMRL
has instituted a temporary policy that requires the full funding of CAs at the time of award. The
review team believes this new policy would inhibit APPCD from awarding broadly worded
agreements with multiple projects since the decision would have to be made at the time of award
as to what projects merit funding. However, the policy was identified as temporary to the review
team;.so this may only help to resolve the problem for the short term. Furthermore those broadly
worded agreements that are still active present a continued opportunity for abuse.
Recommendations:
1' V '
The review team makes the following recommendations:
4.3.1 The Director, APPCD should take appropriate measures to ensure all POs clearly \
understand what changes require the issuance of formal amendments from GAD. Furthermore,
the staff should utilize the expertise of the EMS on all such matters.
4.3.2 The Director, APPCD should take appropriate measures to ensure that assistance
agreements are not being utilized for the purpose of acquiring direct support for the Agency. The
Director must ensure that Division personnel are aware of the different circumstances authorizing
the use of assistance vs. acquisition.
4.3.3 The Director, APPCD should evaluate the practice of calling for post-award work plans
to define the research project(s) in light of the vulnerabilities created. For those still hi existence,
management controls should be developed and implemented within APPCD to scrutinize new
research projects being initiated so as to ensure the research does not represent a direct Agency
need; i.e., one that is immediate, uninterrupted, and specific. At a minimum the EMS should
review and concur with the initiation of all new projects under existing agreements.
4.3.4 The Director, APPCD should conduct an examination of active competitive CA files to
50
-------
determine whether there are existing comprehensive summaries of the competitive process that
simply were not included in the PO's award file. If any are found,, they should be added to the
file. If none are located, we recommend that a summary be prepared, even though it would be
after-the-fact, so that any future reviewers could at least follow the process.
4.3.5 The Director, APPCD should should take appropriate action to ensure that future awards
are more thoroughly documented relative to competition. While this could be done in the
decision memorandum, in most cases it would probably work better to have a separate summary
document, with the relevant RFA, application abstract and peer reviews attached, and with the
approval official's specific rationale. If this approach is used, then the additional documentation
should be cross-referenced in the decision memorandum.
4.3.6 The Director, APPCD should assess the Division's peer review process related to research
proposed under extramural vehicles. The EMS, some of the Division's experienced POs, and
other experts on peer review within ORD (e.g., in NCERQA) should be consulted. If the
Division continues to award assistance agreements that call for the submission of work plans
after award, then the Director should take appropriate measures to ensure those work plans
receive the appropriate level of peer review, in conformance with Agency and ORD policy.
FINDING: Scope Modified without GAD Approval
DIVISION RESPONSE A: On CR822870, the APPCD Project Officer had been discussing
this situation and an amendment to the SOW with the Grants Specialist since the summer of
1996 in conjunction with and while attempting to resolve the controversy between RT1,
the University of Illinois, and PETL. That situation has now been resolved.
• i
ACTION TO BE TAKEN: An amendment to the SOW was requested in November of 1996.
While we have verbally followed up with GAD on that action, the amendment has never been
issued. We will followup in writing and continue to pursue the matter until they act.
DIVISION RESPONSE B: NCERQA effort under CR824152 was completed and we do not
intend to allow any further funding or effort from them under this cooperative agreement.
ACTION TO BE TAKEN: We will monitor this agreement to ensure no future changes take
place without GAD participation.
DIVISION RESPONSE C: Under CR824308 with the University of California, we concur
that such an amendment is needed but we believe that the 95/96 time frame would have been
premature. In view of the dwindling resources, we believe our Project Officer was following a
good management practice in asking the cooperator to limit their efforts to the completion of one
task thus allowing time for EPA and the cooperator to seek additional funding sources for the
other tasks.
51
-------
ACTION TO BETAKEN: We will ensure that an amendment is secured.
FINDING: Improper use of Assistance for Direct Benefit ,,
DIVISION RESPONSE: We concur that there is an appearance of such a situation in some
documents in some files (examples were furnished in 4 of the 17 files reviewed) but we believe
this is primarily due to a lack of sensitivity to appearance regarding this issue by our personnel at
that time (1991 to 1995). The Assistance Project Officers training course which stresses this
issue did not begin until April of 1996. We would also like to point out that one of those four
examples cited mentions quotes from the cooperator, not an APPCD employee. In another
example cited, the quote is from a peer reviewer who seems to have been under a mistaken
impression that the research supported would have some effect on the regulatory phase out of a
chemical by OAR. In reality, the chemical phase out had been determined much earlier by the
Montreal Protocol. The research supported by this agreement is simply to develop more
economical, environmentally friendly alternate refrigerants than those currently available to
industry and users and to the best of our knowledge this effort does not directly benefit EPA.
NO ACTION NECESSARY
RECOMMENDATIONS:
4.3.1 DIVISION RESPONSE: The Review Team recommended that we ensure all P.O.s
understand what changes require formal amendments.
ACTION TO BE TAKEN: We will issue written guidelines to be sure Project Officers
understand what consitutes a change. This guidance will be based on the Assistance
Administration Manual.
4.3.2 DIVISION RESPONSE: The Review Team recommended that we ensure all assistance
agreements are not being utilized to acquire direct agency support.
ACTION TO BE TAKEN: We believe that on the whole our assistance agreements have been
properly chosen; however, we will institute a review of all active Coop's for this purpose.
4.3.3 DIVISION RESPONSE: The Review Team recommended that we evaluate the practice
of calling for post-award work plans-develop and implement management controls to scrutinize
new research projects.
ACTION TO BE TAKEN: We have done so and will limit this practice to only large and
complex projects where it is appropriate to secure work plans for peer review of the specific
details of test protocols, statistical approaches, and similar efforts.
4.3.4 DIVISION RESPONSE: The Review Team recommended that we examine all active
competitive coop files to determine whether comprehensive summaries of the competition exists
that were not included in the P.O. files... If so,add to the file...If not, create such summaries and
52
-------
add to the file.
ACTION TO BE TAKEN: In conjunction with the file review^.mentioned in response to 4.2.1
and 4.3.3, we will review and dpcument as time permits.
4.3.5 DIVISION RESPONSE: The Review Team recommended that we ensure all future
awards are more thoroughly documented relative to competition.
ACTION TO BE TAKEN: Through the checklists discussed earlier and through our file
reviews and approval process, we will ensure future awards are more thoroughly documented
with respect to the competitive process followed, (also see 4.3,6).
4.3.6 DIVISION RESPONSE: The Review Team recommended that we assess APPCD peer
review process and that if post award work plans are continued, we should ensure they are peer
reviewed.
ACTION TO BE TAKEN: We have done so and, as a result, we propose to begin using paid
peer reviewers so that we can better control their schedule and output, Our purchase orders will
specifically require detailed narrative to support numerical scores on the issue of post-award
work plans, see 4.3.3 above.
4.4 INTERAGENCY AGREEMENTS
INTRODUCTION. Nine IAG files were reviewed and five POs for lAGs were interviewed,
listed as follows with the POs who were interviewed marked with an asterisk.
'•i.
IAG NO. AGENCY PROJECT OFFICER
RW64937353 Tennessee Valley Authority Charles Sedman*
DW57937570 U.S. Air Force Theodore Brna
RW69937476 Federal Highway Administration Carl Ripberger
DW89936536 Oakridge National Laboratory Robert Hendricks*
DW80936395 NASA Geddes Ramsey*
DW89937403 DOE/Sandia Laboratories Charles Miller*
RW89936334 Department of Energy Susan ThomeloerHoward
DW49936487 National Science Foundation Christopher Geron*
DW96936393 Corps of Engineers Samuel Rakes
Specific findings and recommendations follow:
1. Some Statements of Work for LAGs Were Well-Developed-Others Not
Findings:
.'' ?
53
-------
The review team found that some of the SOWs for LAGs were well developed while
others were not. Guidance provided in the 1AG Compendium at 4.A-10 reads:
t t
"The scope of work should be clear and comprehensive. It should include responsibilities
and deliverables associated with the agreement."
Current training for IAG POs stresses the importance of well-defined SOWs. Good IAG SOWs
should include details on planned activities, milestones, and deliverables. In addition,
cooperation authority LAGs should address the responsibilities of both agencies.
A. Examples of lAGs with Well Developed SOWs
o IAG DW49935487 had a very well defined SOW that provided detail on the planned
activities, milestones, and expected deliverables.
o IAG DW80936395 with NASA for a paint stripping study had a good SOW. It has 6
specific task areas with specific assignment of duties between EPA and NASA. In addition,
there are modifications that make line-by-line changes as the work evolved and requirements
changed. (However there were other problems with this agreement as noted in section 2 which
follows.)
1 • .1..', •.
B. Examples of lAGs with Less Than Ideal SOWs
o The SOW for IAG DW 96936393 was typed onto the EPA form 1610. The SOW
- described in general terms what the goal of the agreement was and little else. The decision
memorandum provided slightly more detail as to what was planned, but not very much. If
performance under this SOW does not provide what is expected, there is very little recourse for
the EPA to either obtain the expected deliverable or recover funds.
o The SOW for IAG RW64937353 with the TVA was awarded under a cooperative
authority but the SOW is silent on TVA's participation. The PO noted that EPA's in-kind
contribution was to manage the project actually being accomplished by RTI under a CA. He
noted that, under an MOU (not in file), EPA and TVA had been co-developing the technology
for a coal fired boiler but in this case EPA was providing technical expertise for TVA and
"working for them".
o The SOW for IAG RW89936334 with DOE was fairly, specific about the minimum
work that EPA will ensure is completed but provided no time line for the three year agreement,
no cost breakdown, and was silent on DOE's participation.
/
A well defined SOW is critical for ensuring that the Agencies participating in an IAG
understand their respective rights and obligations. It is also critical for developing the cost
estimates as discussed in the following section. LAGs are in essence contracts between two
agencies and as such should be clearly defined to the maximum extent feasible.
54
-------
2. Front-end Budget Development Lacking for New lAGs
Findings;
The review team found that some of the lAGs did not include evidence of the preparation
of an independent Agency cost estimate. The IAG Compendium at 4.V-3 stipulates that the
originating office is responsible for developing a preliminary cost estimate for the project which
considers personnel, travel, equipment, supplies, procurement/assistance, construction, other
costs, and indirect costs. The IAG Compendium further stipulates that the originating office
must negotiate the final terms of the IAG including the final cost estimate. Some examples
where these criteria were not met follow:
o The file for IAG RW89936334 with DOE did not include evidence of an EPA
developed preliminary cost estimate. EPA did not provide DOE with sufficient information to
make a determination of price reasonableness,
o The file for RW64937353 with TV A also lacked an independent EPA preliminary cost
estimate and provided insufficient information to TVA for them to assess the reasonableness of
the cost. The PO noted that EPA had relied on TVA's estimate for the LAG costs but had no
documentation of those costs in the file. Nor was there any breakdown of EPA's $50,000 in-
kind contribution to the project.
o The review team found no preliminary cost estimates developed for lAGs
DW49936487 with NSF arid DW96936393 with COE. There was a general budget breakdown
on the LAG itself. Whether the budgets were fed directly from the other agency or the result of
negotiations cannot be determined from the file.
o The file for LAG DW80936i395 with NASA had no front end budget documentation.
As each increment of funds was added, there was a limited description of the costs that EPA
funds would be spent on, specifically including an equipment list. There appears to be an
understanding that NASA would contribute in-kind resources, but they are never quantified.
There also may be an Air Force contribution, though again it is not clear in the file as to amount
or purpose. Even the amount of EPA's potential obligation is unclear. There are file notes to the
effect that NASA is planning on our obligating between $100K and S200K in each of the three
years. In fact, so far we have only obligated $275K total, and it is not at all clear that there will
be any more funds obligated to the LAG. None-the-less, the PO believes that NASA is incurring
costs on the assumption that additional funding is forthcoming. The PO is aware of the need to
advise NASA not to expect more than what we have to obligate, and will likely reduce the SOW
as necessary. But all parties seem uncertain of what that amount will be, or even of what we
have tentatively promised.
The failure to develop independent preliminary cost estimates for LAGs creates a situation
where EPA is over-reliant on the other Agency's budget estimate which in turn could lead EPA,
55
-------
particularly under funds-out lAGs, to expend funds beyond what is necessary to complete the
work. Failure to develop an independent cost estimate negates an opportunity for EPA personnel
to better understand the full extent of the proposed agreement. t,
Recommendations:
The review team makes the following recommendations:
4.4.1 The Director, APPCD should take appropriate measures to ensure IAG SOWs are clear
and comprehensive, describe the responsibilites of both Agencies, and identify anticipated
deliverables and due dates. i
4.4.2 The Director, APPCD should take appropriate measures to ensure compliance with
Agency policy that calls for the development of an independent Agency budget estimate as an
integral tool for negotiation of the final terms of an IAG.
4.4.1 DIVISION RESPONSE: The Review Team recommended that we ensure IAG
statements of work are clear and comprehensive.
ACTION TO BE TAKEN: We will communicate these requirements to our staff and enforce
them through our review and approval process.
4.4.2 DIVISION RESPONSE: The Review Team recommended that we ensure compliance
with requirements to develop an independent government cost estimate.
i
ACTION TO BE TAKEN: We will communicate these requirements to our staff and enforce
them through our review and approval process.
4.5 CONTRACTS
INTRODUCTION. Seven contract files, one delivery Order file, and three work assignment
files were reviewed. Iii addition, five contract POs, one DOPO, and three WAMs were
interviewed. The contracts/work assignments/delivery order that were reviewed are listed below
with those individuals who were interviewed identified with an asterisk.
CONTRACT NO. CONTRACTOR PROJECT OFFICER
68D10008 International Fuel Cells Corp. Ronald Spiegel*
68D50163 EC/RInc. Charles Mann*
68D20186 International Fuel Cells Corp. Susan Thorneloe-Howard*
68D40123 Dr. Malcolm Leadbetter Judith Ford*
68D30045 Research Triangle Institute Judith Ford
68D40100 E.H. Pechan and Associates Susan Thomeloe-Howard
68D40005 Acurex Corporation Judith Cook*
56
-------
WA 52 P. Limieux (WAM)*
WA30 R.Kulp(WAM)*
WA 49 , N.D. Smith (WAM)*
68W50065 OAO Corporation (FAIR Delivery Order) ' Charlotte Bercegeay*
(DOPO) : ..,
The specific findings and recommendations are discussed as follows:
1. WAM Support to Project Officer under Acurex Could Be Improved.
Findings;
The review team identified some areas where involvement of the WAMs with the PO and
CO on critical contract management activities under the Acurex contract was either lacking or
undocumented. The specific findings are as follows:
o The review team found that the WAMs are not fully supporting the PO under Contract
No. 68-D4-0005. The PO was selected due to her administrative skills as opposed to scientific or
technical qualifications. The review team found little effort being made by the WAMs to review
monthly costs, and no input by them to the PO for her to,use in acting to recommend payment on
invoices. In addition, the WAMs rarely copy the PO on correspondence, meeting
agenda/minutes, or other documentation relative to technical performance. Other than for certain
regular outputs (work plans and monthly reports), it is not clear that either the WAMs or the PO
are closely tracking the receipt of deliverables. The rationale provided for the lack of invoice
input was that the contractor gives little hard data for the WAMs to review. The absence of
technical progress/performance input seems to relate to a sense that the issues are beyond the
field of the PO, coupled with an absence of recognition of the obligation to keep her informed of
such matters.
Subsequent to the review, the APPCD EMS informed the review team that structural
changes were being made at the time of the review. In October, 1996 OAM changed the
reporting requirements of the contract to increase the level of detail provided for costs incurred
on each work assignment. In addition, OAM provided a training session for RTF personnel
which highlighted the changes to the invoicing and financial reporting requirements for
contractors and which explained the invoice review procedures for WAMs and POs. In addition,
the APPCD EMS held follow-up sessions with each branch to go over the invoice review process
in more detail.
i
Pursuant to the Contracts Management Manual Chapter 7, the contract Technical
Direction Clause and the new invoice approval procedures, the role of the WAM is critical for
providing the technical expertise that an administrative type PO doesn't have. This includes
responsibilities ranging from assessing the costs vouchered in comparison to work performed to
assessing the adequacy of deliverables. Accordingly it is critical that the WAMs work closely
with the PO and CO to effectively manage the contract, and that they keep the PO and CO
apprised of their technical monitoring activities such as the issuance of technical direction.
57
-------
Indeed it is in the best interests of the WAM to be able to show that they put the PO/CO on
notice of the directions they give.
< i
Recommendations
The review team makes the following recommendations:
4.5.1 The Director, APPCD should take appropriate measures to determine the effectiveness of
the new procedures for WAM technical support to the PO and address any vulnerabilities which
persist.
4.5.2 The Director, APPCD should take measures to ensure WAMs understand that it is
required by the Technical Direction clause that they give copies of technical directions to both
the PO and the CO. It may be necessary for the PO or the EMS to refresh the WAMs on the
distinctions between Technical Direction and technical communication.
4.5.3 The Director, APPCD should take appropriate measures to ensure deliverables are
recorded and tracked under contracts, and that the review and acceptance of deliverables is
documented in the POs' and WAMs' files.
4.5.1 DIVISION RESPONSE: The Review Team recommended that we determine the
effectiveness of new WAM support and address vulnerabilities.
ACTION TO BE TAKEN: We will do so by periodically discussing this with the APPCD
Project Officers of our level-of-effort, work assignment contracts.
4.5.2 DIVISION RESPONSE: The Review Team recommended that we ensure WAMs
understand the requirement to give copies of technical directions to the Project and Contracting
Officers.
ACTION TAKEN: We have done so by written communications to our staff (see Attachment
6) and will check compliance by periodically consulting with the applicable Project Officers.
i i i
4.5.3 DIVISION RESPONSE: The Review Team recommended that we ensure deliverables are
recorded, tracked and documented.
ACTION TO BE TAKEN: We will do so by distribution of the file checklist discussed earlier
and by random review of files to ensure compliance.
58
-------
EXHIBIT 1-ADDITIONAL DETAIL RELATED TO CR822870
BACKGROUND
CR822870 serves to illustrate a number of findings contained in this report; namely that
some of the assistance agreements are being utilized to acquire direct support for the Agency,
that peer review of the actual research being performed is lacking, and that some of the effort is
outside the scope of the approved agreement. This CA, with a start date of 8/1/94, was awarded
to Research Triangle Institute (RTI) for the purpose of improving indoor air quality-'To develop
and demonstrate engineering approaches hi buildings systems that will improve IAQ hi
residential and nonindustrial commercial buildings." RTFs proposal identified thirteen
individual tasks that they would undertake. The CA called for the recipient and EPA to
cooperate in the development of research plans for the specific tasks after award.
EXAMPLE OF ASSISTANCE UTILIZED FOR DIRECT SUPPORT
j*-'
Section 3.2 of RTFs proposal is entitled "Ventilation Systems Research" and includes the
following sections: _
3.2.1 Introduction
3.2.2 Strategies to Prevent or Minimize Adverse HVAC Impact on IAQ—
The stated concept is to conduct laboratory and field investigations to develop innovative
HVAC design, operation, and maintenance practices to prevent or minimize adverse IAQ impact
on HVAC systems. An extensive rationale is then presented discussing various problems with
indoor air attributed to HVAC systems. The proposed approach is to conduct a laboratory and
field study of HVAC systems as sources of indoor air contaminants. The first year was proposed
to evaluate the problem through a literature search followed by an engineering analysis of the
problem areas. The proposal notes that "The answers will not all be available in the literature,
and field studies of the conditions at the problem locations will probably be necessary (emphasis
added) to complete the engineering analysis." Further on the proposal states: "Most of the effort
will be placed in the review and the field studies, with a small bench-scale laboratory effort if
required (emphasis added). " As stated hi the proposal, the second and third years of the research
program will be focused on developing remediative and preventive strategies. The description of
the proposed research actvities is generic; e.g., it states "building problem areas will be
reproduced in the laboratory at an appropriate scale to serve as test beds for improvement." No
further details are provided. At the end of the section it describes one aspect of cooperation as
the involvement of key EPA personnel in planning and defining the specific research subtasks.
By memorandum dated 3/16/95, one of the Division's engineers submitted a request to
the Project Officer to "...initiate the issuance (of) a work assignment to Research Triangle
-------
EXHIBIT 1
Institute under the above Cooperative Agreement. The scope of the research is shown in the
attached Scope of Work (SOW) and is designated a (sic) Phases I and ffl." The SOW is 9 pages
in length and includes Appendix A (5 pages) with task descriptipns and Appendix B (3 pages)
describing HAC System Components. The Introduction notes the SOW was developed by the
Project Officer to provide "...general information and descriptions that convey .the research
proposed by the Environmental Protection Agency." It further states that the research project
responds "...to a request by the Program Office for information, data and guidance concerning
the use of air duct cleaning (ADC) for the maintenance of acceptable indoor air quality (IAQ) in
residential dwellings." (The EPA engineer sponsoring this project was interviewed as a WAM
under the Acurex contract. During the interview he confirmed this project was undertaken at the
request of the Office of Air and Radiation.) Finally the Introduction indicates the research
described in the SOW "...was developed from research proposals by associates currently
performing research for AEERL"; namely Acurex, RTI, the U. of Minnesota and the U. of
Illinois.
The project was divided into four phases with RTI having the lead for Phases I and ffl
and Acurex (APPCD's on-site support contractor) having the lead for Phases n and IV. Phase I
of the project calls for RTI to design and construct a residential HAC system pilot-scale test
setup. In Appendix A, it states for this task that "RTI shall develop the design criteria and the
design drawings with assistance from Acurex." Under Phase n, Acurex "will evaluate and select
the methods and procedures that will be used in the HAC system pilot-scale test setup. Under
Phase El, RTI is to evaluate the selected tests using the test setup. In Phase IV, Acurex will
design and perform a field study using up to nine occupied homes. Outputs 1,2 and 5 are peer
reviewed papers, output 3 is a guidance document of recommended practices, and output 5 is a
peer reviewed EPA report. The SOW did not make it clear as to which organization was
responsible for which output. In Appendix A, Task 1-1 calls for submission of a work plan for
PO approval which ".i.will show how the Contractors plan to accomplish the work including
schedules, timeliness, milestones, and outputs." •••''',.
While the research described in EPA's SOW falls within the generic research proposed
by RTI under Section 3.2 of its proposal, the documentation supports the conclusion that the
primary purpose of this particular research project is for the direct support of the Agency. The
key factors leading to this conclusion include:
o The need for this specific research project was identified by an Agency program office.
o The Agency provided the specifications for this project.
o Deliverables include an EPA guidance document and an EPA report.
LACK OF PEER REVIEW FOR RESEARCH PROJECT
Another problem associated with this particular research project is the lack of peer review
for this effort. The original RTI proposal leading to the award of the cooperative agreement did
-------
EXHIBIT 1
undergo peer review, but there is no evidence that this particular project received a similar level
of peer review. The peer reviews conducted on the original proposal lacked substance with the
comments addressing the capabilities of RTI to perform the typess of research projects that were
described generically with little technical detail. Only one peer reviewer critiqued any of the
technical aspects of the proposal, and the critique was limited to portions of 3 distinct projects,
though not the project identified here. An effective peer review of the this specific research
project would have afforded the Agency an opportunity to strengthen the scientific merit of the
proposed research.
EXAMPLE OF WORK PERFORMED OUTSIDE THE ORIGINAL SCOPE
RTI's originally approved research narrative at pages 3-44 to 3-46 described a research
effort entitled "Numerical Modeling of Ventilation Effectiveness" which included a number of
subtasks to "...build on current modeling capabilities and validation studies to make the existing
model more flexible and more representative of real situations." Subtask 1 calls for
incorporation of a broad range of source/sink models into the model. The approach for this
subtask was for RTI to conduct a complete literature survey to update the model through
incorporation of all possible source/sink characteristics. The key personnel proposed to conduct
this research effort were identified as RTI employees.
Subtask 4 (pages 3-51 thru 3-53) of this same section of RTI's proposal was entitled
Model Validation Study with the concept being to validate the predictive power of developed
spatial and temporal models to ensure applicability to real indoor air situations. In the approach
it notes that the testing of the current 2D model will be continued collaboratively between RTI
and the Bioenvironmental Engineering Research Laboratory at the University of Illinois (UI)
with UI providing the laboratory test facilities in which to construct the test rooms and assemble
the HV AC system, etc. Along with RTT personnel, one of the two key UI participants was
identified as Dr. Leslie Christiansen.
Approximately in June, 1995 APPCD was notified that one of its proposed research
projects under the Environmental Technology Initiative (ETI) had been accepted. The stated
purpose was to create a test facility to characterize the chemical and paniculate emissions of
materials and products used hi ulterior environments. APPCD's partners for this project
included the Corps of Engineers Construction Engineering Research Laboratory (CERL), the
Illinois Technology Center, and the Bioenvironmental Environmental Research Laboratory at UI.
The proposal indicated that $800K would be provided via an existing cooperative agreement to
UI, though it did not indicate which vehicle. It also indicated $100K would be provided to RTI
via a CA, though it did not indicate here whether this was a new or existing CA. The abstract for
this ETI project reads as follows:
"The objective of this proposal is to accelerate the development of green products for use
in indoor environments. This will be accomplished through the establishment of a 'state
of the art' test facility to characterize the indoor performance parameters (e.g., chemical
profiles, emission factors, modeled exposure potential) of new low emitting/low impact
products. The facility will be operated by a not-for-profit entity (i.e., The University of
-------
EXHIBIT 1
Illinois) and will provide direct, low cost support to affected market segments, such as
paints, aerosol products, electronics, etc. Commercial users will be given essential design
and performance data to guide in the development of low emitting/low impact products
for use (in) interior environments. AEERL and CERL will provide long term guidance
and support in methods development, development of baseline data, and resolution of
QA/QC issues."
The first task was to develop an approved work plan/business plan within 3 months after
the start of the project. Ultimately the facility would be developed and tested so as to allow the
initiation of commercial operations within 24 months from start-up. The cost for the first two
years was estimated at $950K. The key Ul participant was identified as Dr. Leslie Christiansen.
APPCD determined that the ETI project could be incorporated under CA CR822870 with
RTI. However, the ETI project was delayed for an extended period of time due to UI's inability
to develop an acceptable work plan. Ul, however, proceeded to incur costs. By October 1996,
UI had submitted invoices for about $300K, most of which were disallowed by the EPA PO. For
example, $100K of subcontractor costs was disallowed because the subcontractor had not been
approved for use under the CA. Subsequently UI solicited support from an Illinois Congressman
to intercede on its behalf. The EPA PO was requested to provide background information for
ORD management which he did by memorandum dated 10/7/96. Among other things, the PO
indicated the following in his memo:
/
"When APPCD was notified of the award it was apparent that the project could not be
implemented unless an existing cooperative agreement were used.
Research Triangle Institute (RTF) and the University of Illinois were partners in a
cooperative agreement that was awarded to RTI as part of APPCD's competitive
cooperative agreement process. This cooperative agreement called for RTI to be the lead
and for the University of Illinois to be a subcontractor for research .that EPA, RTI, and the
University agreed on. Product emission testing was part of the scope of this cooperative
agreement. A meeting between RTI, the University, and EPA was held to determine if
the verification project could be implemented as part of this cooperative agreement. All
parties agreed that the proposed verification project was the type of project intended to be
implemented under the cooperative agreement. It was decided to implement the
verification project under the EPA/RTI cooperative agreement."
Shortly after the project began, Dr. Christiansen notified everyone that he felt that
because there were numerous labs doing emission testing but with differenct protocols,
that a new lab should not be established but rather effort should be put into:
Developing and adopting standard test protocols
Bringing the various stakeholder (manufacturers, designers, building owners,
laboratories) together
Everyone agreed that this revision in effort would improve the program.
-------
EXHIBIT 1
RTI, with EPA approval, awarded the university of Illinois a contract under the
cooperative agreement. ..."
< i
The review team does not agree with the decision to include the ETI project in the CA
with RTI. Subsequent to the review, the review team was advised that the section of RTI's
proposal under which this ETI project was incorporated was Subtask 1 under 3.4.2. While there
may be some link between the ETI project and the goal of incorporating "...a broad range of
source/sink models into the model", RTI's proposal narrative does not contain any hint that this
would entail the establishment of a testing lab as described above. Furthermore it was Subtask 4
rather than Subtask 1 that identified participation of UI under this CA. The review team does not
believe Subtask 4 bears any clear relationship to the ETI proposal either. The review team
concludes that this ETI project is outside the scope of the CA and was processed under the CA
for expediency sake because the ETI funds were awarded too late in FY 95 to process a new
award. The review team believes CA 822870 was selected because it had a broadly stated
purpose of product testing and because UI had been identified as a subcontractor under one of the
subtasks.
As a side note, the review team questions the decision to modify this project as indicated
in the PO's memo of 10/7/96. The ETI proposals were selected after a rigorous competition that
included peer review. This project was selected on the basis of UI establishing a testing lab for
indoor air products. That concept has apparently been replaced with an entirely new focus. The
review team is uncertain of the procedures for changing the essence of the approved research
projects; but it would seem logical that such an extensive change would require some sort of
approval from the organization that had overall responsibility for ETI; i.e., the Office of Policy,
Planning and Evaluation.
-------
EXHIBIT 2
- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
'i NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
° AIR POLLUTION PREVENTION AND CONTROL DIVISION
RESEARCH TRIANGLE PARK, NC 27711
OFFICE OF
MAY 061997 RESEARCH AND DEVELOPMENT
nse to APMageme
raft Report
TO: Sandy Croley,
Management Review Team Leader, NRMRL
Attached is APPCD's response to the Management Review that your team
conducted here last October. We have thoroughly reviewed the report, and - on the
whole - we are pleased with our performance. However, I recognize that some significant
deficiencies were identified, particularly in the area of the appearance of improper use of
cooperative agreements and our less-than-ideal documentation of the peer review process
for such agreements. As you can see in our attached response, we have, or will shortly
have, instituted constructive procedural changes to deal with these and other deficiencies
identified.
Thank you and your team for the time and effort you devoted to this activity. We
trust that we will make an even better showing next time as a result of your
recommendations.
Attachment (1)
cc: Tim Oppelt
Cal Lawrence
Management Review Team
Recycled/Recyclable • Printed with Vegetable Oil Based Inks on 100% Recyded Paper (40% Postconsumer)
-------
EXHIBIT 2
CHAPTER 3 - FINANCIAL MANAGEMENT
3.2 FUNDS CONTROL
3.2 2 REVIEW OBSERVATIONS & FINDINGS
RESPONSE: AOS did not feel that it was necessary to include the Object Classes (OC) in the
extramural Document Control Register (DCR) since the extramural expenditures were limited to
only a few OCs. When APPCD's internal data base was established for FY96, we did not
recognize the importance of including and tracking by, the OMIS project numbers in the
extramural DCR. In FY97 APPCD modified the extramural automated DCR to include the
object class and the OMIS project number for each commitment/obligation.
RESPONSE: AOS does not receive all obligating documents to input in the DCR. We will
make every effort to obtain all obligating documents once we have reviewed the MARS Report.
ACTION TAKEN: All obligations have been entered in the extramural DCR via award
documents and shall be kept up-to-date.
3.2.3 RECOMMENDATIONS
.RESPONSE: It is difficult from the Management Review Draft Report to know what DCR had
the discrepancies, therefore, both the intramural and extramural OCRs were reviewed.
ACTION TAKEN: Both the intramural and extramural OCRs have been checked for any,
discrepancies and, when necessary, corrections made.
RESPONSE: The review team recommends the use of OMIS which would support the project
tracking that is needed to correct any DCR omissions.
ACTION TAKEN: We are continuing to input into OMIS, run reports, and obtain training on all
modules of OMIS, which will help us is many different ways, including project tracking. In FY96
we ran into so many barriers and unknowns that it was difficult to rely on the outputs of the data.
We also continued to run into problems that only the OMIS HOTLINE could resolve, I believe
that the staff is more aware of the capabilities of OMIS and plan to use it to its full extent.
3.3 TRAVEL
332 REVIEW OBSERVATIONS & FINDINGS
RESPONSE: The report states: "The funding for all travel is recorded in the Document Control
Register before the approving official signs." This is incorrect. The approving official signs all
-------
EXHIBIT 2
TAs prior to recording in the DCR.
3 3.3 RECOMMENDATIONS
RESPONSE 1: AOS contacted the branch's support staff and determined that their files
contained copies of receipts to support travel documents. It is not the responsibility of the FCO
to keep receipts.
NO ACTION NECESSARY
RESPONSE 2: The TA does not require dates of signatures but this is a requirement for
vouchers.
ACTION TAKEN: AOS sent out a memo to recommending and approving officials to ensure
signatures on TAs and vouchers are dated. (See Attachment 1)
RESPONSE 3: To authorize use ofPOV as the primary mode of transportation does require a
special justification that proves it is more advantageous to the government. However, there is an
exception. According to Federal Travel Regulation 41 CFR Part 301-4.2, a special justification is
not required for round-trip by POV in lieu of taxi between common carrier terminal and office or
residence. Our local travel office and travel training we have received, support this position. We
were instructed that the 'X' on the POV line in Block #11 is sufficient without providing a
justification in the Remarks block. We verified this information with GSA.
NO ACTION NECESSARY
' - '
3.4 TIMEKEEPING
3.4.7 RECOMMENDATIONS
RESPONSE A: Timekeepers' copies of T&As do not include the supervisors' signature unless
the supervisor makes a copy for the timekeeper before hand delivering to the DA.
ACTION TAKEN: The DA has offered to the supervisors to make copies of the signed iimecards
for the timekeepers if they request. (See Attachment 2)
B. NO RESPONSE OR ACTION NECESSARY.
C. NO RESPONSE OR ACTION NECESSARY.
3.5 BANKCARD
/
3512 PROCUREMENT REQUEST
-------
EXHIBIT 2
RECOMMENDATION
^^~^™^^~*™ ™™^™^"^~*"™~fc— . - f
RESPONSE: Based on required training for bankcard holders and approving officials, it is not
required to acquire signatures by the Approving Official for each individual item on the log sheet
or on a Procurement Request (EPA 1900-8). Signature on the log sheet at the end of the cycle is
sufficient. AOS confirmed this with Alan Ritter, the Bankcard Program Trainer. (See Attachment
3) ' .;(y
NO ACTION NECESSARY.
3.5.1.3 RECONCILED STATEMENT OF ACCOUNT
RESPONSE: Bankcard Holders or Approving Official did not have copies of the back page of
the statement with the Approving Officials dated signature. The Team recommended that copies
of this page be copied after signatures.
i
ACTION TO BE TAKEN: AOS will send out a memo recommending that the Approving
Officials make a copy of the back page of the statement to be kept by the Bankcard Holder.
RESPONSE: The Review Team recommends that the log sheets and statements include a
COMPLETE but brief description of item(s).
ACTION TO BE TAKEN: AOS will send out a memo with the Review Team's recommendation.
RESPONSE: The Review Team indicated that portable electric power tools are listed as
prohibited items.
ACTION TAKEN: AOS contacted the Bankcard Program Coordinator and this item was
removed from the restricted list in 1995. (See Attachment 3)
3.5.1.4 SUPPORTING RECEIPTS
RESPONSE: The Review Team suggests that the Bankcard Holders remind the merchant,
before the purchase is made, that the purchase is TAX EXEMPT. v
ACTION TO BE TAKEN: AOS will send out a memo reminding Bankcard Holders that all
purchases are tax exempt.
3.5.1.5 BANKCARD LOG
RESPONSE: The Review Team noted that the "Date Ordered" and "Date Received" blocks on
the bankcard log sheets were not always complete.
-------
EXHIBIT 2
ACTION TO BE TAKEN: AOS will send out a memo reminding Bankcard Holders to submit a
complete log sheet including the date ordered block and date received block, if item(s) have been
received prior to the end of the billing period. "
3517 SUMMARY/RECOMMENDATIONS
RESPONSE: The Review Team was unable to obtain copies of authorized signatures by the
approving officials or the FCO on the Bankcard Logs.
ACTION TAKEN: The FCO sends the Bankcard Holder a copy of his/her log sheet with
signatures and accounting data.
RESPONSE: Bankcard Holders were not aware that ALL documentation must be kept for at
least one full year.
ACTION TO BE TAKEN: AOS will send out a reminder regarding the length of bankcard
documentation retention.
RESPONSE: The Review Team could not determine whether the statement was submitted to
the Cincinnati FMC within the 5 day period and suggested that the statement be date stamped
when received by the Bankcard Holder. This is not a requirement.
• i ~
ACTION TO BE TAKEN: AOS will send out a memo with this suggestion.
-------
EXHIBIT 2
4.2 CROSS CUTTING ISSUES
GENERAL RESPONSE TO CROSS CUTTING FINDINGS: We note that much of the ORD
guidance for file documentation and Decision Memorandum content came into being after the
award of the actions reviewed. For example, most of the actions reviewed were awarded before
the issuance of the July, 1994, version of Chapter 12 of the ORD Policy and Procedures Manual.
We also note that a number of the items discussed above under "decision memos..." are still not
covered by the 1995 version of Chapter 4 of that Manual (e.g., there is no guidance regarding
discussion of substantial involvement, competition and travel), In any event, please be assured
that we plan to improve the quality of our documentation and files as recommended and we
propose the following actions in response to the recommendations:
RECOMMENDATIONS
4.2.1 RESPONSE: The Review Team recommended that we ensure all open extramural files
reflect all required approvals.
ACTION TAKEN: We are instituting a program to review all of our active extramural files and
we will obtain any missing documents as time permits.
t
4.2.2 RESPONSE: The Review Team recommended that we ensure approvals are obtained and
documentation is retained for all future actions...all files are well organized and include evidence
of monitoring.
ACTION TAKEN: We are preparing file checklists for various types of actions (competitive and
non-competitive cooperative agreements, reimbursement and disbursement lAG's) and, after
coordinating with NRMRL, we will distribute these and provide assistance to the APPCD Project
Officers in utilizing them. We believe that this, along with a diligent review and documentation of
the approval process for future actions, will solve much of this perceived problem.
4.2.3 RESPONSE: The Review Team recommended that we strengthen decision memos.
ACTION TAKEN: In March of 1996, we issued guidance and a standard format (see
Attachment 4) for decision memos for both cooperative agreements and lAGs based on the
requirements of Chapter 4 of the ORD Policy and Procedures Manual and the text for the ORD
Project Officer's Assistance Course. We believe that this, along with a diligent review and
documentation of the approval process will adequately deal with this issue.
4.2.4 RESPONSE: The Review Team recommended that we ensure future coops and lAGs
require detailed financial reporting.
ACTION TAKEN: In the past, the financial reporting requirements have been specified by GAD.
However, in December of 1996, we issued guidance regarding terms and conditions for
cooperative agreements, including examples of financial and technical status report requirements
(see Attachment 5). In addition, we will attempt, during the above mentioned file reviews, to
-------
EXHIBIT 2
identify needs and to initiate amendments where appropriate to obtain more detailed financial
reporting, within the limits set by 40 CFR 31.40, 40 CFR 160, and OMB Circular A-110 Part 50.
* <
4.3 COOPERATIVE AGREEMENTS
FINDING: Scope Modified without GAD Approval . ,
RESPONSE A: On CR822870, the APPCD Project Officer had been discussing this situation and
an amendment to the SOW with the Grants Specialist since the summer of 1996 in conjunction
with and while attempting to resolve the controversy between RTI, the University of Illinois, and
PETL. That situation has now been resolved.
ACTION TO BE TAKEN: An amendment to the SOW was requested in November of 1996.
While we have verbally followed up with GAD on that action, the amendment Has never been
issued. We will followup in writing and continue to pursue the matter until they act.
RESPONSE B: NCERQA effort under CR824152 was completed and we do not intend to allow
any further funding or effort from them under this cooperative agreement.
ACTION TO BE TAKEN: We will monitor this agreement to ensure no future changes take
place without GAD participation.
RESPONSE C: Under CR824308 with the University of California, we concur that such an
amendment is needed but we believe that the 95/96 time frame would have been premature. In
view of the dwindling resources, we believe our Project Officer was following a good
management practice in asking the cooperator to limit their efforts to the completion of one task
thus allowing time for EPA and the cooperator to seek additional funding sources for the other
tasks. '
ACTION TO BE TAKEN: We will ensure that an amendment is secured.
FINDING: Improper use of Assistance for Direct Benefit
RESPONSE: We concur that there is an appearance of such a situation in some documents in
some files (examples were furnished in 4 of the 17 files reviewed) but we believe this is primarily
due to a lack of sensitivity to appearance regarding this issue by our personnel at that time (1991
to 1995). The Assistance Project Officers training course which stresses this issue did not begin
until April of 1996. We would also like to point out that one of those four examples cited
mentions quotes from the cooperator, not an APPCD employee. In another example cited, the
quote is from a peer reviewer who seems to have been under a mistaken impression that the
research supported would have some effect on the regulatory phase out of a chemical by OAR. In
reality, the chemical phase out had been determined much earlier by the Montreal Protocol. The
research supported by this agreement is simply to develop more economical, environmentally
friendly alternate refrigerants than those currently available to industry and users and to the best
of our knowledge this effort does not directly benefit EPA.
-------
EXHIBIT 2
NO ACTION NECESSARY
RECOMMENDATIONS:
4.3.1 RESPONSE: The Review Team recommended that we ensure all P.O.s understand what
changes require formal amendments.
ACTION TO BE TAKEN: We will issue written guidelines to be sure Project Officers
understand what consitutes a change. This guidance will be based on the Assistance
Administration Manual.
4.3.2 RESPONSE: The Review Team recommended that we ensure all assistance agreements
are not being utilized to acquire direct agency support.
ACTION TO BE TAKEN: We believe that on the whole our assistance agreements have been
properly chosen; however, we will institute a review of all active Coop's for this purpose.
4.3.3 RESPONSE: The Review Team recommended that we evaluate the practice of calling for
post-award work plans,,.develop and implement management controls to scrutinize new research
projects.
ACTION TO BE TAKEN: We have done so and will limit this practice to only large and
complex projects where it is appropriate to secure work plans for peer review of the specific
details of test protocols, statistical approaches, and similar efforts.
4.3.4 RESPONSE: The Review Team recommended that we examine all active competitive
coop files to determine whether comprehensive summaries of the competition exists that were not
included in the P.O. files... If so add to the file...If not, create such summaries and add to the file.
ACTION TO BE TAKEN: In conjunction with the file reviews mentioned in response to 4.2.1
and 4.3.3, we will review and document as time permits.
4.3.5 RESPONSE: The Review Team recommended that we ensure all future awards are more
thoroughly documented relative to competition,
ACTION TO BE TAKEN: Through the checklists discussed earlier and through our file reviews
and approval process, we will ensure future awards are more thoroughly documented with respect
to the competitive process followed, (also see 4.3.6).
i !
4.3.6 RESPONSE: The Review Team recommended that we assess APPCD peer review process
and that if post award work plans are continued, we should ensure they are peer reviewed.
ACTION TO BE TAKEN: We have done so and, as a result, we propose to begin using paid
peer reviewers so that we can better control their schedule and output. Our purchase orders will
specifically require detailed narrative to support numerical scores on the issue of post-award work
-------
EXHIBIT 2
plans, see 4.3.3 above.
4.4 INTERAGENCY AGREEMENTS;
RECOMMENDATIONS:
4.4.1 RESPONSE: The Review Team recommended that we ensure LAG statements of work are
clear and comprehensive.
ACTION TO BE TAKEN: We will communicate these requirements to our staff and enforce
them through our review and approval process.
4.4.2 RESPONSE: The Review Team recommended that we ensure compliance with
requirements to develop an independent government cost estimate.
ACTION TO BE TAKEN: We will communicate these requirements to our staff and enforce
them through our review and approval process.
4.S CONTRACTS: ,,-
v '. •
RECOMMENDATIONS:
./
4.5.1 RESPONSE: The Review Team recommended that we determine the effectiveness of new
WAM support and address vulnerabilities.
ACTION TO BE TAKEN: We will do so by periodically discussing this with the APPCD Project
Officers of our level-of-effort, work assignment contracts.
4.5.2 RESPONSE: The Review Team recommended that we ensure WAMs understand the
requirement to give copies of technical directions to the Project and Contracting Officers:
ACTION TAKEN: We have done so by written communications to our staff (see Attachment 6)
and will check compliance by periodically consulting with the applicable Project Officers.
4.5.3 RESPONSE: The Review Team recommended that we ensure deliverables are recorded,
tracked and documented.
ACTION TO BE TAKEN: We will do so by distribution of the file checklist discussed earlier and
by random review of files to ensure compliance.
-------
EXHIBIT 2 Attachment 1
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
AIR POLLUTION PREVENTION AND CONTROL DIVISION
RESEARCH TRIANGLE PARK, NC 27711
251997
MEMORANDUM RESEARCH AND DEVELOPMENT
SUBJECT: Travel Procedure
FROM: DeanieGagnon •
Chief, AOS
TO: Frank Princiotta
Branch Chiefs
This is to request that all signatures, on both the travel authorization and the travel
vouchers, be dated (even though this is not required on the TA). Also, please remind travelers
that vouchers must be submitted within seven days after returning from the trip.
If you have any questions please let me know.
cc: Kathrine Morgan
Blair Martin
-------
EXHIBIT 2 Attachment 2
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
AIR POLLUTION PREVENTION AND CONTROL DIVISION
. RESEARCH TRIANGLE PARK, NO 27711
APR 2 5 1997
OFFICE OF
RESEARCH AND DEVELOPMENT
SUBJECT: Timekeeping
FROM: Brenda Massengill
Designated Agent
TO: APPCD. Branch Chiefs
During the NRMRL Management Review last October, APPCD's timekeeping process was
reviewed to determine if Agency procedures are being appropriately and effectively implemented.
I would like to take this opportunity to address some key points to ensure that we continue to
follow established guidelines.
Please hand deliver all timecards to me in a sealed envelope. This includes retro cards. I realize
, that there are rare instances when you must have someone else hand deliver them to me, but they
MUST be in a sealed envelope. If you need me to provide a copy of the signed timecards to^your
timekeeper, please let me know.
Some of you have been submitting the timecards as early as Wednesday morning. I realize this
may be necessary on occasion due to travel, compressed day off, or scheduled leave. However,
the need for many last minute changes and retro cards will be eliminated if you submit them closer
to the deadline. Timecards are due to me by 9:00 a.m. on Friday unless I have notified you and
the timekeeper that an early submission is required. In those instances, I will let you and the
timekeeper know in advance when they must be submitted.
Thank you for making the timecard process an easy one for me. All of our timekeepers are aware
of the Agency policies and procedures on timekeeping and they do their job very well. We rarely
have corrections or problems with the timecards as submitted.
Please call me if you have any questions.
cc: Frank Princiotta
Deanie Gagnon
Blair Martin
APPCD Timekeepers
-------
— INU t) (-j
07:26 EPfl'OflrvPTOD'ICb * 319 541 I5j,6
EXHIBIT 2 Attachment 3
From: ALAN RITTER
TO : RTPMAINHUB . INTERNET . MORGAN-KATHRINE
Date: 4/14/97 7: 20am
Subject: BANKCARD APPROVALS
1. While the procedures (dated 1993, updated via announcements
1993-97) do state the Approving Official must approve all
purchases, these approvals are not req'd on an individual items
basis if the cardholder and the Approving Official have an agreed
to list of approved purchases and the cardholder only buys things
from the list. (IF PURCHASES ARE REQUIRED THAT ARE NOT ON THE
APPROVED LIST, THEN THE A.O. WOULD BE PROPER TO REQUIRE UP-FRONT
APPROVALS FOR THIS NON-STANDARD ITEMS.)
< •>
Many times things like buying office supplies are inherent in
ovn's job description, so buying office supplies does not require
individual approvals unless the A.O. so imposes this requirement.
Otherwise, one signature on the log is sufficient.
2. A brief description on the items purchased has always been
required on the statement of account before it is sent to the
Cine. Fin. Mgt. Office. THERE HAS NOT BEEN ANY RELEASE OF THIS
REQUIREMENT AND SUCH HAS NEVER BEEN STATED IN MY TRAINING.
3. Power Hand Tools (prior schedule number 51 VI) is indeed
non-mandatory, and has been since early 1995.
If you/ others still have questions, please call/write/email/fax
me your inquiries.
-------
EXHIBIT 2
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
AIR POLLUTION PREVENTION AND CONTROL DIVISION
RESEARCH TRIANGLE PARK, NC 27711
ATTACHMENT 4
Date:
March 29, 1996
OFFICE OF
RESEARCH AND DEVELOPMENT
SUBJECT: Decis^iori^Me^randafoj: Interagency Agreements
^ ~ b—^ "—"""^
FROM: Mai. Huneyetffft~, "Extramural Specialist
TO:
Mai. Hune;
All APPCD Project Officers
ORD has developed a standard format for Decision Memo's for
Interagency Agreements. This standard format is attached for
your information. A similar one for Cooperative Agreements
undoubtedly soon will be issued but hasn't as of this date.
Please note that earlier this year, we were instructed to use; a
standard heading for all Decision Memo's as follows:
'Date:
Subject:
From:
To:
Proposed (Disbursement) (Reimbursement!
Interagency Agreement between EPA and
Frank T. Princiotta (MD-60)
Recommending Official, APPCD
E. Timothy Oppelt (MD-235)
Decision Official, NRMRL"
' Please note that we need to show Frank Princiotta as the
Recommending Official and Tim Oppelt as the Decision Official on
these documents as well as on Decision Memo's for Cooperative
Agreements, and on Funding Orders.^ On future actions, please do
not show Branch Chiefs as the Recommending Official nor Frank as
the Decision Official. Frank can sign for Tim on some actions
and Branch Chiefs for Frank, but even in these cases,(mostly
incremental funding actions) we should show Frank as Recommending
Official and Tim as Decision Official. Neither Millie Lee nor
Scott McMoran should be shown on a Decision Memo. Normally a
transmittal memo will not be required. (
I'm sorry about all the changes and all the confusion and, I
apologize for any part of it I've contributed to so far. It does
seem to me though that things are settling down both in Contracts
and Grants. But who knows, maybe it's just that we're in the
lull before the storm. Call me on Ext. 2903 if I can help.
ii... u<:l<«l/lloc:v:l.it)!(j • <•&:'•• ; ,-.•'!• V s'cil.iMi: Oil M.r.'M ink,. 111. ion-. H.-.V lull f'.iiJi-r Mir !•.>••,'. ,>M-.im«T>
-------
EXHIBIT 2
ATTACHMENT A
CONTENTS OF DECISION MEMORANDA FOR IAGS
A.
B.
D.
Background/
Objectives
Justifica-
tion
Relationship 1,
of Project to
Program Mission
Specific Pro- 1,
gram I.D.
Planned Pro- 1,
ject Outputs
Explanation of 1,
any apparent
duplicative or
excessive effort
G. Quality Assur- 1
ance
2
H. Optional 1
1. Provide sufficient detail for Decision
official to understand the proposed action.
2. Identify duration and overall budget,
including out years.
3.. Cite authority and explain why applicable.
4. Certify proposed IAG is consistent with
EPA Program For Office's mission.
*
1. For Economy Act:
a. Explain servicing Agency's unique ex-
pertise.
b. Include statement that services cannot
be obtained as cheaply/conveniently thru
commercial' sources. Summarize
alternatives other than IAG that were
considered.
2. For Statutory Authority, describe briefly
each/Agency's role/responsibility.
3. For both, discuss budget detail and include
discussion on reasonableness. Cite
source of funding and if incrementally
funded, when funds are anticipated to be
added to the IAG.
1. Self-explanatory
Relate to program/project planning system
as captured in OMIS.
Could be reports, publications, etc.
Provide statement plus how you arrived at
that conclusion.
Discuss applicability of QA/QC to pro-
ject .
Note review by QA Officer, if applicable.
Summairy of peer reviews and reconcilia-
tion of contrary reviews—if required.
Justification for on-site performance at
EPA facility--SRO approval required per
-------
EXHIBIT 2 ATTACHMENT 5
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
' NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
AIR POLLUTION PREVENTION AND CONTROL DIVISION
RESEARCH TRIANGLE PARK, NC 27711
December 26, 1996
OFFICE OF
•MEMORANDUM : RESEARCH AND DEVELOPMENT
SUBJECT: Terms and Conditions for Cooperative Agreements
FROM: Mai Huneycutt
'Extramural Management Specialist
TO: All APPCD Project Officers
During our recent management review, there appeared to be
some confusion on what we should include as "special terms and x,
conditions" in cooperative agreements. I promised myself that
when things slowed down a little, I would check this subject out
and try to clarify it for those who might be writing new
cooperative agreements. I have found that what Grants refers to
as "special terms and conditions" fall into two categories.
\
First there are the administrative type terms found in Title
40, Code of Federal Regulations, Part 40. These deal with such
matters as the method of payment, procedures for refunds, limits
on rates paid to consultants, requirements for using recycled
paper, goals for subcontracting minimum percentages to small
disadvantaged businesses, the Fly America Act requirements, the
Government Printing Office regulations, reporting on any lobbying
of congress, drug free workplace activities, etc. We don't need
to request that Grants include these terms in our cooperative
agreements nor can we change 'these requirements in any way .since
they are required by law or regulation. ;Grants should include
the latest version of all applicable administrative terms in all
cooperative agreements.
Second, there are the technical type terms such as quality
assurance, requirements for consulting with the EPA Project
Officer at key points in performance, the need for peer reviews,
etc. Some of these technical terms are ORD mandated requirements
and others are just common sense. The largest category of
technical terms involves reporting, such as research plans,
progress reports, financial status reports, invention reports,
equipment reports, final reports, etc. I was once told by Grants
that they automatically include standard reporting requirements
in all. cooperative agreements. It turns out they have NOT always
done this. Even when they did, their standard progress reports
were on a quarterly basis since this is the shortest interval for
-------
EXHIBIT 2 ATTACHMENT 5
TERMS AND CONDITIONS
i •
COOPERATIVE AGREEMENT No. CR
1. REPORTING REQUIREMENTS - The recipient will prepare and
submit the reports described herein. (Five) copies of each
report will be submitted to the designated EPA Project Officer -
and one copy to the Grants Specialist-
la ) _Res£cLrch_PXaji - A plan for .conducting the research will be
submitted to the EPA Project Officer within 30 days after the
date of this cooperative.agreement. The plan must be consistent
with the application and must be mutually acceptable to the
parties before the research begins Bunder the project. The plan
will show individual tasks and the relationship between the tasks
as well as the personnel assigned, budget and schedule for each
task. <
The recipient will prepare revisions to the research plan
when the requirements are changed by amendment to the agreement
or when requested by either the principal investigator or project
officer. The recipient may submit recommended revisions to the
plan if the recipient believes such revisions are desirable for
achievement of project objectives. All revisions to the plan
must be concurred with by the EPA Project Officer before the
revised effort begins. One copy of each revision to th.e plan
will be provided to the Grants Specialist.
(b) Quality Assurance Plan. - A Quality Assurance Plan, consistent
with the application and other terms of this agreement will be
prepared by the recipient and submitted, to the EPA Project
Officer no later than 30 days before any data collection
activities are scheduled to begin. A copy will be provided to
the Grants Specialist. The level of detail in the QA Plan
depends on the QA category that was assigned to the project by
the EPA Project Officer, and the QA Category is dependent on the
anticipated end use of the data:
o Category I - Regulatory, enforcement, legal or major
policy matters
o Category II - Establishment of rules, regulations or
policies > '
o Category III - Evaluation of control technologies
o Category "IV - Feasibility studies or fundamental
investigations
-------
ATTACHMENT 5
EXHIBIT 2
Environmental Protection Agency is required. In addition:
i . An annual invention statement is re'quired with each
continuation application;
,ii. A final invention report is required within 90 days
after completion of the project period; and
iii. When a Principal Investigator changes institutions or
ceases to direct a project, an invention statement must
be promptly submitted with a listing of a 1.1 inventions
during his administration of the agreement.
(e) Equipment Report - At the completion or termination of a
project, the recipient will prepare and-submit to the EPA Project
Officer, a listing of all items of equipment acquired with EPA
funding with an acquisition cost of $1,000 or more and having a
useful life of more than one year.
i ' '
(f) Final Report - The recipient will prepare and submit to the
EPA Project Officer, a draft of the final report for review no
later than 90 days prior to the end of the'approved project
period. The report will document project activities over the
entire period of the agreement and shall describe the recipient's
achievements with respect, to the stated project purposes and
objectives. The report will set forth in complete detail, all
technical aspects of the project, both negative and positive,
recipient's findings, conclusions, and results, and including, as
applicable, an evaluation of the technical effectiveness and
economic feasibility of the methods or techniques investigated.
The final report will include EPA comments when-provided by
the Project Officer. Prior the end of the project period, one
hard copy reproducible master, suitable for printing, and one
copy on disc, (other copies may also be stipulated in the
agreement) will be provided to the Project Officer. One copy
shall be provided to the Grants Specialist. The disc supplied by
the recipient shall be produced with an Microsoft Disk Operating
System (MS-DOS) or a Personal Computer Disk Operating System (PC-
DOS),, version 2.0 or higher. Files must be in ASCIIs(American
Standard Code for Information Interchange). The disks must be 5
1/4 inch double-sided/double-density disks and they must include
both the final report and the project summary.
7 REPORT FORMATS - All documents produced for publication under
this agreement shall be in accordance with the requirements set
forth in 40 CFR 40.160-5 and the "Handbook for Preparing Office
of Research and Development Reports" dated 8/95 or such later-
version in effect at the time of report preparation.
-------
EXHIBIT 2 ATTACHMENT 5
b. Organization Char; : Identify pro'ect and QA personnel,
titles and areas of responsibility for performance of work.
(It is highly desirable that the QA function be, independent
of project groups that generate data.)
c. QA Personnel Authority and Responsibilities: Describe
the authority and responsibility of QA personnel who will
implement the organization's Quality Assurance Plan. Also
describe procedures for assuring that necessary corrective
actions are implemented.
d. QA Experience: Identify types and degrees of experience
in developing and .applying QA procedures to data generating
activities proposed for the cooperative agreement.
e. QA Personnel Experience: Describe background and
experience of the proposed personnel relevant to the
accomplishment of the proposed QA Program.
-j
5. FEDERAL INVOLVEMENT - This assistance agreement has been
classified as a cooperative agreement, indicating there will be
substantial federal involvement. EPA's substantial involvement
will include (a) providing technical and quality assurance
review; (b) providing information from earlier studies of .a
similar nature; (c) facilitating communications between research
cooperators, federal state and local agencies interested in the
results of this project; (d)... ,
6. KEY PERSONNEL - The recipient's key personnel who will
participate in the research under this cooperative agreement are:
(include individuals, who will charge 20% or more of
their time to the agreement). >
7. MEETINGS - Occasional meetings will be held at times and
sites mutually agreed to by the parties for the purpose of review
of research accomplished, interpretation of results, planning for
future research,.. and whenever deemed appropriate by either
party to this agreement.
-------
EXHIBIT 2 ATTACHMENT 6
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
NATIONAL RISK MANAGEMENT RESEARCH LABORATORY
AIR POLLUTION PREVENTION AND CONTROL DIVISION
RESEARCH TRIANGLE PARK, NC 27711
APR 2 41997 RESEARCH AND DEVELOPMENT
MEMORANDUM:
SUBJECT: Technical T^ffiietion Oo4h.e(Acurex Contract
FROM:
TO: All Work Assignment Managers
Based on information from the Management Review, I am very concerned that
Work Assignment Managers are not providing the Project Officer, Judy Cook, and the
Contracting Officer with copies of technical direction that is issued to Acufex. Please
understand that the issue is not whether or not the P.O. and the CO. will understand what
is contained in the technical direction, but rather the issue is that we are not adhering to a
specific requirement included in the contract - a requirement which is intended to ensure
that the P.O. and the C.O. have a complete file on each Work Assignment. In Section
H.15(e) the contract states that "Technical direction will be issued in writing or confirmed
in writing within five calendar days after verbal issuance. One copy of the technical
direction memorandum will be forwarded to the Contracting Officer and the Project
Officer."
In the future, please ensure that a copy of all technical direction is sent to the
Project Officer and the Contracting Officer for 68D40005. Thank you for your
cooperation.
cc: Blair Martin
DeanieGagnon
Branch Chiefs
Reeve led/Recyclable . Pnnlcd with VegclaWe O.| Base* Inks on 100% Recycled Pap*' '«•/, Po«cor>sum*.|
-------
EXHIBIT 3-1
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
5 NATIONAL RISK MANAGEMENT RESEARCH LABORATORY'
5 CINCINNATI, OH 45268
June 6, 1997
OFFICE OF
RESEARCH AND DEVELOPMENT
MEMORANDUM
SUBJECT: Evaluation of APPCD's Response to Financial
Management Review, Section 3 of the Draft
ORD Management Review Report (March 1997)
FROM: Sandy Croley, Overall Team Leader
APPCD Management Review Team
TO: Draft Report
After careful review of APPCD's response to the Management Review Draft Report,
the Review Team for the Financial Management portion of the review have the following
comments. ,Vy-
Overall, APPCD's response was acceptable and the action indicated appropriate. There
are two exceptions the Review Team Members would like to address.
3.3.3 Travel Recommendations
APPCD Response to #3. "To authorize use of POV as the primary mode of
transportation does require a special justification that proves it is mpre advantageous to the
government. However, there is an exception. According to Federal Travel Regulation 41
CFR Part 301-4.2, a special justification is not required for round-trip by POV in lieu of taxi
between common carrier terminal and office or residence. Our local travel office and travel
training we have received, support this position. We were instructed that the "X" on the POV
line in block #11 is sufficient without providing a justification in the Remarks block. We
verified this with GSA."
f '
The Management Review Team has the following comment. While the Federal Travel
Regulations quoted above are true, EPA Travel Regulations (RMD 2550B Exhibit 3-3, and
RMD Chapter 4, para 5.h) states that authorization of POV and rental car must be in the
Remarks block of the TA - not just in the funding blocks. Furthermore, EPA Travel
regulations always take precedence over any Federal policies imposed. EPA Travel
Regulations are derived directly from the Federal Travel Regulations with additional EPA
imposed rules. What GSA requires has no bearing on what EPA requires. (Note: The
Management Review Team Members are concerned with the different guidance being
disseminate from the various Travel offices (Finance). We would like to recommend this
Recycled/Recyclable .Printed with Vegetable Oil Based Inks on 100% Recycled Paper (40% Postconsumer)
-------
EXHIBIT 3-1
issue be address by ORD Senior Management in coordination with Senior Management in
OARM. The different guidance and/or rules being followed continues to be an issue with
ORD Laboratories that have several Servicing Finance Offices.)
Section 3.5.1.2 Bankcard Procurement Request. APPCD's response was as follows:
"Based on required training for Bankcard holders and approving officials, it is not
required to acquire signatures by the Approving Officials for each individual item on the log
sheet or on a Procurement Request (EPA 1900-8). Signature on the log sheet at the end of the
cycle is sufficient. AOS confirmed this with Alan Ritter, the Bankcard Programer Trainer. NO
ACTION NECESSARY".
Alan Ritter states (LAN message to {Catherine Morgan 4-14-97), "....While the procedures
(dated 1993, updated via announcements 1993-97) do state the Approving Official must approve
all purchases, these approvals are not req'd on an individual item basis if the Cardholder and the
Approving Official have agreed to a list of approved purchases and the cardholder only buys
things from the list. (IF PURCHASES ARE REQUIRED THAT ARE NOT ON THE
APPROVED LIST, THEN THE A.O. WOULD BE PROPER TO REQUIRE UP-FRONT
APPROVALS FOR THESE NON-STANDARD ITEMS)". Alan goes on to state, "Many times
things like buying office supplies are inherent in one's own job description, so buying office
supplies does not require individual approvals unless the A.O. so imposes this requirement.
Otherwise, one signature at the end of the cycle, 17th of the month on the log is sufficient".
\ - -
M ' /
The Review Team Members take exception to this statement. This is a direct
contradiction to what is stated in the materials supplied to review team members. (Management
Review Guidelines and Revised EPA Guidelines For Use Of Government Bankcards (August,
1996). Of major importance is the issue that APPCD did not obtain any approvals for bankcard
purchases except for ADP.
/
Information given to the reviewers during the interview was that the Bankcard Holder and
Approving Official did not have knowledge of obtaining approvals before purchases were made
(except ADP). If Alan Ritter is correct about a LIST, this was never supplied or mentioned to the
reviewers by the Bankcard Holders or Approving Officials when questions were raised to them
about prior approvals.
At a minimum, the list should be attached to the logsheet to verify mat the
purchases were on a pre-approved list. Otherwise, one would conclude these purchases were
unauthorized. The list would also have to be very explicit as to total number of allowable items
to be purchased. (Note: This is an ongoing concern within ORD. The team members have
researched the issue and have found that new Guidelines for Use of Government Bankcards is
currently being revised. However, the new Bankcard policies and procedures still DO NOT
reflect how the Management Review Guidelines are written.)
-------
EXHIBIT 3-2
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
'*i-~acP OFFICE OF RESEARCH AND DEVELOPMENT
NATIONAL EXPOSURE RESEARCH LABORATORY
CINCINNATI, OH 45268
MEMORANDUM
DATE: July 2, 1997
SUBJECT: Evaluation of APPCD's Response to the to Section 4 of the Draft ORD
Management Review Report (March 1997)
FROM: Walter R Stutts, Extramural Team Leader
APPCD Management Review Team
TO: Sandy Croley, Overall Team Leader '
APPCD Management Review Team
The extramural review team evaluated the APPCD response to the extramural portion
(Section 4) of the draft management review report. The consensus of the extramural review team
relative to the Division's response is set forth below:
OVERALL RESPONSE:
Generally, there is a lack of a plan to document the process and report the results to NRMRL
management. There is little indication of who will do what and when they will do it. As a result,
accountability is vague except that the Division Director will see that the effort is completed. A
better response would have included an action plan describing exactly what would be done, who
has the lead, and when the effort will be completed so that NRMRL management can effectively
track the progress. Responses to most IG reports provide a template for such action plans. (See
for example the Action Plan included in the IG Report No. E6ABG5-04-0110-6400045 entitled
"Special Review - Allegations of Improper Use of EPA Facilities at the Gulf Ecology Division,
National Health and Environmental Effects Laboratory".)
RESPONSE TO INDIVIDUAL FINDINGS AND RECOMMENDATIONS:
4.2.1
The details of the program are sketchy relative to how the effort will be undertaken and how
the results will be documented. The use of the term "as time permits" may not result in any firm
action. The team recommends that the Division make a firm time commitment for a defined
action plan.
4.2.2
The team does not believe that checklists alone will guarantee adequate monitoring, though
they should facilitate the process. The team notes that the Division's response is unclear as to
-------
EXHIBIT 3-2
how assistance is to be provided to POs on the use of the checklists; i.e., who will provide, how,
and when? "Diligent review" is nebulous—what steps are being taken to strengthen the current
review process?
4.2.3
The review team considers this response to be a good step, but'the Division should explain
how they intend to document this action and the Division should make a firm time commitment
for the review. The team also recommends that the Division revisit the March 1996 guidance in
light of new guidelines issued by GAD in September 1996. Again, the team considers the term
"diligent review" to be nebulous.
4.2.4
The team does not consider the phrases "we will attempt" and "where appropriate" to be
firm commitments. Whose judgement determines "where appropriate" if no standard is set? The
Division should establish a specific date for the completion of this review and-update.
4.2.5
There was no Division response to this recommendation. The response to 4.2.4 may have
been intended to respond to 4.2.5 at the same time; If so, the response did not fully address the
recommendations in 4.2.5. For example, does the Division plan to reemphasize theiinancial
responsibilities of POs unde'r assistance agreements and if so, how?
• 'r-' .
4.3 FINDINGS-Scopes Modified Without GAD Approval:
CR822870
The response is unclear as to what was amended. The draft report cites a number of
shortcomings relative to this CA. Did the amendment require sole-source processing? The team
recommends the Division provide more detailed information to enable NRMRL management to
assess the adequacy of the actions taken. Furthermore, if the amendment has not yet been issued,
the team recommends the Division elevate the issue.to ORMA and/or to the appropriate Branch
Chief in GAD for assistance.
CR824308
The team questions the validity of the Division claiming that the PO was "following a good
management practice in asking the cooperator to limit their efforts to the completion of one task
thus allowing time for EPA and the cooperator to seek additional funding sources for the other
tasks." This is a cooperative agreement, not a contract. The cooperator is not doing the work
for us. The PO should not be "managing" the effort; rather he should be "monitoring" it. This
illustrates the problem with these large, generally defined coops having the appearance of being
used to provide direct support to the Agency. If the scope needed to be decreased, it would be
up to the recipient to propose how to do that. Why would EPA be seeking additional funding
sources for the other tasks? Again, it looks like direct support.
-------
EXHIBIT 3-2
4.3 FINDINGS—Improper Use of Assistance for Direct Support
The response appears to attempt to downplay the significance of the finding. The concern
regarding direct support is still an issue under CR824152, and CR824257 listed 8 examples with
only three of them occurring during the period that personnel were "insensitive to the
appearance." The significance of the Division's statement that one of the examples quotes the
cooperator instead of an EPA employee is unclear. If the cooperatbr believes the work is at the
direction of and for the direct benefit of the Agency, then it would appear to support the
conclusion that the work performed was improper for the type of instrument. While the team will
not dispute the Division's response relative to the Texas A&M CA, theother findings certainly
lead to a firm conclusion that the work performed was for EP A's direct benefit.
4.3.1
The response fails to address the role of the EMS. Will written guidance suffice, or is some
remedial training in order?
4.3.2
The team notes that there is little detail provided. How will the review be conducted, when,
and by whom? How will the results of the review be documented? The team does not believe
that the second part of the recommendation was addressed; i.e., how does the Division plan to
make the staff aware of the issue? In addition, the team recommends that the section entitled
ACTION TO BE TAKEN be expanded to include an increased emphasis on the discussion of
principal purpose in Decision Memoranda in the future so as to fully establish the support and
stimulation aspect of the proposed assistance agreement.
4.3.3
The team feels the Division's response missed the point. The examples that the team cited in
the draft report were not "large and complex projects" per se: rather they were generally defined
areas for research cooperation which were definitized after funding became available and a
particular need was identified by the Agency. Furthermore, the response would appear to
contradict NRMRL's recently issued Guidelines for Extramural Agreements which identified
broadly worded cooperative agreements that call for post-award work plans as a vulnerability and
which calls for assistance proposals with "a clearly defined scope and objectives". Finally, the
response is limited to new awards and fails to address the initiation of tasks under existing open-
ended agreements. The team recommends the involvement of the EMS in the review of these
"new starts" (including the review of any work plans) to ensure that the past abuses are not
repeated.
4.3.4
..i •
Again, this response lacks detail as to who is going to do this review, in what manner, and
during what time frame. The use of the phrase "as time permits" may not result in any firm
action.
-------
4.3.5 EXHIBIT 3-2
The team believes that the use of checklists alone will not likely resolve the issue and
recommends that some guidelines and perhaps training are needed to set the standard for
documentation of competition. The team provided some specific recommendations, but it is
unclear whether the Division considered those recommendations.
/ •
4.3.6 ;
I ' ' '
The team recommends that the Division apprise NEMRL management of how they proceeded
and how they documented the results. Furthermore, the Division needs to address the peer
review of work plans generated after award.
4.4.1
It is unclear how the Division plans to communicate the requirements to the staff. More detail
on the "review and approval" process would help solidify the Division's oversight plan.
i '
4.4.2
The review team had the same reaction as outlined in 4.4.1. In addition, the team found
during the interview process that many of the POs said that they didn't know how to do IGCEs
and that is why they didn't do them. Accordingly the team recommends the ACTION TO BE
TAKEN be expanded to include training on IGCEs for POs.
4.5.1
The response is unclear as to what mechanism will be used on a periodic basis. Will this be
done monthly, yearly, or on some other periodic basis?
4.5.2
The Division failed to address the other part of the recommendation relative to clarifying the
difference between TD and TC.
4.5.3
It is unclear as to who will have responsibility for conducting the random reviews. The team
recommends that the Division consider involving the Technical Information Manager in
recording, tracking and documenting deliverables.
We hope these observations are useful to you and your management. If you have any
questions, please give me a call. I have asked the other extramural team members to forward
their notes and papers to me so that I can consolidate them prior to providing them to you for
your archives. Once that task is completed, we consider our participation in the APPCD review
to have been fulfilled.
-------
AIR POLLUTION PREVENTION AND CONTROL DIVISION
Frank PrincoUa. DirecUx
Kit Chappell, Secretary
Blair Martin
Ann Barbee, Secretary
Mai Huneycutl
Mike Maxwell
Pat Sharpe
ADMINISTRATIVE OPERATIONS STAFF
Deanie Gagnon. Chief
[Vacant, Secretary]
Judy Cook
Larry Farmer
Joyce Fowler
Brenda Massengill
Kalhrine Morgan
Bill Whelan
EMISSIONS CHARACTERIZATION
AND PREVENTION BRANCH
Larry Jones, Chief
[Vacant, Secretary]
Chuck Darvin
Chris Geron
Bruce Harris
Julian Jones
Sue Kimbrough
Foy King, Jr.
Mike Kosusko
Chuck Mann
Chuck Masser
Bob McCrillis
Carlos Nunez
Geddes Ramsey
Ted Ripberger
Niranjan Vescio
ChelVogel
KayeWhHfield
AIR POLLUTION TECHNOLOGY BRANCH
Bob Hall. Chief
Jean Gunn, Secretary
Brian Gullett
Norm Kaplan
Jim Kilgroe
Dave Lachapelle
C. W. Lee
Paul Lernieux
Bill Linak
Andy Miller
Dick Ogan
Sam Rakes
Charlie Sedman
Richard Valentine
Jack Wasser
EPA EMPLOYEES
TECHNICAL SERVICES BRANCH
Wade Ponder. Chief
[Vacant. Secretary]
Nancy Adams
Charlotte Bercegeay
Judy Ford
Wayne Fowler
Jim Montgomery
Lynn Pendertraft
Jeff Ryan
Richard Shores
Shirley Wasson
Vacant (ADP)
ATMOSPHERIC PROTECTION BRANCH
Bill Rhodes, Chief
[Vacant, Secretary]
Evelyn Baskin
Lee Beck
Bob Borgwardt
Ted Bma
Cynthia Gage
Bob Hendriks
Jim Jetter
Dave Kirchgessner
Carol Purvis
Dean Smith
Ron Spiegel
Susan Thornetoe
INDOOR ENVIRONMENT
MANAGEMENT BRANCH
Mike Osborne, Chief
Gerri Collins-Lynch, Secretary
John Chang
Bruce Henschel
Betsy Howard
Russ Kulp
Kelly Leovic
Mark Mason
Marc Menetrez
Ron Mosley
Richard Perry
David Sanchez
Les Sparks
Ray Steiber
Gene Tucker
James White
APPENDIX A
-------
AIR POLLUTION PREVENTION & CONTROL DIVISION
MANAGEMENT REVIEW TEAM
OCTOBER 20 - 25,1996
Sandy Croley
Team Leader
NRMRL
Cincinnati, OH
William Bailey
OARM
Cincinnati, OH
Susan Campbell
NRMRL
John Cline
NERL
Larry Hodgson
NHEERL
Linda lori
NRMRL
Stephen Kovash
NRMRL
SueMcKenzie
NRMRL
CarolAnn Sakiewicz
NHEERL
Linda Schwaergerle
NCEA
WaltStutts
OARM/NERL
Cincinnati, OH
RTF, NC
Gulf Breeze, FL
Cincinnati, OH
Ada, OK
Cincinnati, OH
Narragansett, RI
Cincinnati, OH
Cincinnati, OH
APPENDIX B
-------
8/30/96.
LIST OF INDIVIDUALS WITH DELEGATED AUTHORITY
1. Signing extramural resource documents ^
a. Contract incremental funding
Originator /
Branch Chiefs
Project Officer
Funds Certifying Official
b. Small Purchases (with extramural dollars <$50.0K)
Originator
Branch Chiefs
Funds Certifying Official
c. Assistance agreements
Originator
Branch Chiefs
Division Director
Funds Certifying Official
Lab Director
2. Funds Certification
Kathrine Morgan, FCO
Joyce Fowler, Alternate FCO
3. Reprogrammings
Originator (Brenda Massengill)
AOS (Alice C. Gagnon)
Deputy Lab Director
4. Bankcard
a. Bankcard Program Coordinator
Kathrine Morgan
APPENDIX C
-------
b. Bankcard Holders/Approving Officials
Bankcard Holder Approving Official
G. Blair Martin
G Blair Martin
Alice Gagnon
Alice Gagnon
William Rhodes
William Rhodes
Robert Hall
Robert Hall
Larry Jones
Larry Jones
Larry Jones
Michael Osbome
Michael Osborne
Michael Osbome
Wade Ponder
Wade Ponder
Wade Ponder
Branch
IOD
IOD
AOS
AOS
APB
APB
APTB
APTB
ECPB
ECPB
ECPB
DEMB
1EMB
ffiMB
TSB
TSB
TSB
Ann Barbee
Kathleen Chappcll
Brenda Massengill
William Whelan"
Robert Borgwardt
Cynthia Gage
, Paul Lemieux
Richard Valentine
D. Bruce Harris
Charles Mann
Geddes Ramsey
Geraldine Collins-Lynch
MarcMenetrez
Leslie Sparks
James Montgomery*
Lynn Pendergraft
Jeff Ryan
*$20,000
Small Purchases (with expense dollars)
Originator
Branch Chiefs
Funds Certifmg Official
Travel Authorization
Branch Chiefs for staff as Recommending Officer,
Division Director for Branch Chiefs as Recommending Officer
Laboratory Director for Division Director as Recommending Officer
Chief, Administrative Operations Staff as Authorization Officer
Imprest Funds
Not applicable since the Finance Center is located at'RTP. The procedure is the
same as a small purchase order.
Quality Assurance
APPCD's Quality Assurance Officer, Nancy Adams
APPENDIX C
------- |