OAQPS& Office of Air Quality Planning and Standards 2gj»nd stav^united States Environmental Protection Agency Quality Management Plan United StatesEnvironmental Protection Agency Office of Air Quaility Planning and Standards Research Triangle Park NC 27711 ------- Quality Management Plan for The Office of Air Quality Planning And Standards Office of Air Quality Planning and Standards United States Environmental Protection Agency MD-10 Research Triangle Park, NC 27711 11 ------- OAQPS QA MANAGEMENT PLAN APPROVALS sxSh&ture- Joleph B. Elkins, Jr. Qdality Assurance Managei Oflfice of Air Quality Planning and Standards (OAQPS) 1-2C- DATE 7/t,C -/ft SIGNATURE William F. Hunt, Jr. Director, Emissions, Monitoring, and Analysis Division, OAQPS DATE JAtfURE SalAy Shaver Director, Air Quality Strategies and Standards Division, OAQPS SIGNATURE 'Bruce C. Joi'&an Director, Emission Standards Division, OAQPS 7/"/^ DATE SIGNATURE DATE Robert Kellam , , , Acting Director, Infomation Transfer and Program Integration Division, OAQPS SIGNAT^jjSx John S/ Seitz ^ Director, Office of Air Quality Planning and Standards Office of Air and Radiation SIGNATURE Courtney Riordan AA Quality Assurance Representative Office of Air and Radiation DATE & ft / $ & DATE SIGNATURE Nancy' W. Wentworth Director, Quality Assurance Division National Center for Extramural Research Office of Research and Development DATE ' 111 ------- OAQPS QA MANAGEMENT PLAN APPROVALS (Continued) q-in~9b SIGNATURE Peter Wi jPreus^J Ph.D. DirectorV Natidj/al Center for Environmental Research and Quality Assurance Office of Research and Development iv ------- Contents Section Page Revision Date Approval Page Contents Figures and Tables Acknowledgments Acronyms and Abbreviations 1. Management and Organization 1.1 QA Policy Statement 1.2 Organization Chart 1.3 Key OAQPS Personnel 1.4 Technical Activities Supported by the Quality System 2. OAQPS Quality System and Description 2.1 OAQPS QA Application 3. Personnel Qualification and Training 3.1 Training Policy 3.2 Mandatory Training 3.3 QA Training 4. Procurement of Items and Services 4.1 Procurement of Items 4.2 Procurement of services 5. Documentation and Records Management 5.1 Elements of the System 5.2 Document Preparation, Review, and Approval 1/12 3/12 5/12 9/12 1/6 1/3 1/3 2/3 1/3 1/3 1/2 2/2 01 6/96 01 01 01 01 6/96 6/96 5/96 6/96 6. Computer Hardware and Software 6.1 OAQPS Information Management System 6.2 Hardware and Software Requirements 6.3 Data Standards 7. Quality Planning 7.1 Ofiice-wide Planning 7.2 Program-specific Planning 7.3 Project-level Planning 8. Implementation of Work Processes 8.1 Program Implementation 8.2 Project Implementation 1/2 1/2 2/2 1/3 1/3 1/3 1/2 1/2 01 01 6/96 6/96 V ------- Section 9 Assessment and Response 9.1 Management Systems Reviews 9.2 Surveillance 9.3 Technical Systems Audits 9.4 Audit Plan 9.5 Performance Evaluations 9.6 Audits of Data Quality 9.7 Peer Review 9.8 Readiness Review 9.9 Data Quality Assessment 10. Quality Improvement 10.1 Program Review and Improvement 10.2 Project Reviews and Improvement Appendices Page Revision Date 01 6/96 1/8 2/8 3/8 3/8 5/8 6/8 6/8 7/8 7/8 1/2 1/2 01 6/96 A. Group Functions B. Contract QA Form and Guidance C. QAPP Review Form vi ------- Figures Number Title Section/Page 1.1 OAQPS organization chart 1/2 1.2 OAQPS QA Team organization structure 1/6 1.3 Air quality management process 1/8 Tables 2-1 QAPP Elements Applicable to Various Categories 2/5 3-1 Suggested Sequence of Core QA-related Ambient Air and 3/2 Emission Training Courses for OAQPS QA Ambient Air Monitoring Contacts and QA Managers 5-1 QA Documents for Document/Record Storage 5/2 vii ------- Acknowledgments This project would not have been initiated without the original management support and sage advice of Jim Weigold and Iva Spons. The QA Team ap predated the encouragement throughout this project from John Seitz and Bill Hunt. The quality of the document is largely attributable to the OAQPS QA Team and particularly to those team members that v olunteered to be work group leaders. The work group leaders for the Sections were Joe Elkins(l), Michael Papp(2), Randy Waite(3), Mary Jane Clark(4), Doris Maxwell(5), Rhonda Thompson(6), Gar y McAlister(7), Lara Autry(8), David Musick (9), and Terry Harrison(lO). Other OAQPS QA Team member s assisting were Dolores Graf, Mark Schmidt, and Maurice Walker. We are i ndebted to Whit Joyner for professionally editing this document. viii ------- Acronyms and Abbreviations AIRS Aerometric Information Retrieval System APTI Air Pollution Training Institute AQSSD Air Quality Strategies and Standards Division AWMA Air and Waste Management Association CAA Clean Air Act CBI confidential business information CFR Code of Federal Regulations CMD Contracts Management Division CO Contracting Officer DCO Document Control Officer DD Division Director DQA data quality assessment DQAO Deputy QA Officers DQOs data quality objectives EDO environmental data operation EMAD Emissions, Monitoring, and Analysis Division EPA Environmental Protection Agency EPAAR EPA Acquisition Regulations ESD Emission Standards Division ETSD Enterprise Technology Services Division FAR Federal Acquisition Regulations FIPS Federal Information Processing Standards GIS geographical information systems GLP good laboratory practice HAP hazardous air pollutants IAG interagency agreement IDP Individual Development Plans IT information technology ITPID Information Transfer and Program Integration Division LAN local area network MACT Maximum Achievable Control Technology MQAG Monitoring and Quality Assurance Group MQOs measurement quality objectives MSR management system review NAAQS National Ambient Air Quality Standards NAMS national air monitoring station NESHAP National Emission Standards for Hazardous Air Pollutants NIST National Institute of Standards and Technology NSPS New Source Performance Standard OAQPS Office of Air Quality Planning and Standards OARM Office of Administration and Resources Management OIRM Office of Information Resources Management OMB Office of Management and Budget ORD Office of Research and Development PAMS Photochemical Assessment Monitoring Stations PC personal computer PE performance evaluation PR procurement request PSD Prevention of Significant Deterioration QA quality assurance QA/QC quality assurance/quality control QAARWP quality assurance annual report and work plan QAD EPA Quality Assurance Division QAM quality assurance manager IX ------- QAO quality assurance officer QAPP quality assurance project plan QMP quality management plan RCRA Resource Conservation and Recovery Act SCG Source Characterization Group SIPS State Implementation Plans SIRMO servicing information resources management officer SLAMS state and local monitoring stations SOP standard operating procedure SOW statement or scope of work SPMS special purpose monitoring stations SYSOP system operator TQM Total Quality Management TSA technical system audit WAM Work Assignment Manager X ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page: 1 of 12 1. Management and Organization Agency policy initiated by the Administrator in memoranda of May 30, 1979 and June 14, 1979 requires participation in a centrally-managed Quality Assurance (QA) program by all Environmental Protection Agency (EPA) Laboratories, Program Offices, Regional Offices, and by those monitoring and measurement efforts supported or mandated through contracts, regulations, or other formalized agreements. The Agency's policy and program requirements to implement the mandatory QA program are contained in EPA Order 5360.1. The Office of Research and Development (ORD) is responsible for developing, coordinating, and directing implementation of the Agency QA Program. The ORD has delegated this responsibility to its QA Division (QAD) in the National Center for Extramural Research and Quality Assurance. In order to document the adherence to EPA Order 5360.1, EPA requires each organizational unit to develop a Quality Management Plan (QMP) per the specifications in EPA Requirements for Quality Management Plans, EPA QA R-2 Interim Final, August 1994. The QMP is a formal document describing the management policies, objectives, principles, organizational authority, responsibilities, accountability, and implementation plan of an agency, organization, or laboratory for ensuring quality in its products and utility to its users. To implement Agency policy, EPA Laboratories, Program Offices, and Regional Offices are required to prepare a Quality Management Plan (QMP) covering all intramural and extramural monitoring and measurement activities that generate and process environmental data for Agency use. This document contains the Office of Air Quality Planning and Standards (OAQPS) QMP, which delineates the policy and management structure to be used in implementing the Agency QA program and the requirements for preparing individual QA project plans. The purpose of this management plan is to establish a centralized QA program for OAQPS consistent with QA policies and procedures established by the Agency, whose goal is to ensure that data generated by or for OAQPS are of known and acceptable quality. 1.1 QA Policy Statement It is the policy of OAQPS that within the constraints of available resources, QA activities shall be conducted to assure environmental data generated, processed, or used for its program requirements will be of known quality and will achieve prescribed data quality objectives. Furthermore, the data will be adequate and sufficient for their intended use. To ensure that this QA policy is uniformly applied to the generation and processing of OAQPS environmental measurements, the OAQPS QA Manager shall be delegated the authority and responsibility for the OAQPS QMP. The authority covers intramural and extramural environmental data collection and processing as a result of: a. OAQPS in-house environmental measurement activity b. Contracts and Interagency Agreements c. Grants and Cooperative Agreements d. Partnerships with Industry, state and local offices, and Regional Offices An EDO is defined as "work performed to obtain, use, or report information pertaining to environmental processes and conditions," as defined by The American National Standard-Specifications and Guidelines for Environmental Data Collection and Environmental Technology Programs (ANSI/ASQC-E4- 1994). EPA is adopting much of this QA standard in its revised EPA Order 5360.1 ------- Immediate Office Lydia Wegman-Deputy Director Jim Weigold- Assistant Director John Bachmann-Associate Director John Seitz Director Emission Monitoring and Analysis Division Bill Hunt Air Quality Trends Analysis Group Monitoring & Quality Assurance Group Air Quality Modeling Group Emissions Inventory and Factors Group Source Characterization A Group Source Characterization B Group Emission Standards Division Bruce Jordan Metals Group Mineral & Inorganic Chemicals Group Combustion Group Coatings & Consumer Products Group Organic Chemicals Group Waste & Chemical Processes Group Policy, Planning, & Standards Group Figure 1.1. OAQPS organization chart OAQPS Section 1 Revision: 01 Date: 06/21/96 Page: 2 of 12 Wash. Operations Staff Policy Analysis and Communications Staff ] Planning Resources & Regional Management Staff Air Quality Strategies and Standards Division Sally Shaver Integrated Policy & Strategies Group Ozone Policy & Strategies Group Health Effects & Standards Group Risk and Exposure Assessment Group Visibility and Ecosystem Protection Group Innovative Strategies & Economics Group r Information Transfer & Program Integration Division Bob Kellam Operating Permits Group Integrated Implementation Group Information Management Group Information Transfer Group Education & Outreach Group Program Review Group ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 3 of 12 1.2 Organization Chart OAQPS is organized into four Divisions: Emissions, Monitoring, and Analysis Division (EMAD); Emission Standards Division (ESD); Air Quality Strategies & Standards Division (AQSSD); and Information Transfer & Program Integration Division (ITPID). This QMP reflects OAQPS as described in Figure 1.1. The OAQPS will have a QA Manager and each Division will have at least one Deputy QA Officer. 1.2.1 Office of Air Quality Planning and Standards OAQPS is the organization charged under the authority of the Clean Air Act (CAA) to protect and enhance the quality of the nation's air resources. OAQPS sets standards for pollutants considered harmful to public health or welfare and, in cooperation with EPA's regional offices and the states, enforces compliance with the standards through State Implementation Plans and regulations (MACT, NSPS, etc.) that control emissions from stationary sources. The Office evaluates the need to regulate potential air pollutants and develops national emission standards; works with state and local agencies to develop plans for meeting these standards; monitors national air quality trends and maintains a database of information on air pollution and controls; provides technical guidance and training on air pollution control; and monitors compliance with air pollution standards. The Office of Air Quality Planning and Standards, under the supervision of the Director, is responsible to the Assistant Administrator for the specific air quality management and emission standards functions of the Office of Air and Radiation. Specifically, the Office is responsible for: Developing national standards for air quality, emissions standards for new stationary sources, and emission standards for hazardous air pollutants Developing national programs, technical policies, regulations, guidelines, and criteria for air pollution control Evaluating and continuously improving the national air pollution control program and evaluating success in achieving air quality goals Providing a wide range of technical information and support to other EPA Offices, Regions, the nation's state and local agencies, and industry to assist the implementation of air pollution control programs Managing the development and transfer of air program information used to implement air programs and evaluate progress 1.2.2 Emissions, Monitoring, and Analysis Division The Emissions, Monitoring, and Analysis Division (EMAD), under the supervision of its Director, is responsible for directing a national program of scientific and technical policy and guidance for EPA Headquarters and Regional Offices and state and local agencies in air quality monitoring and modeling, control strategy demonstrations, and emissions measurement. In particular, the Division: develops and distributes guidelines for air quality models and provides technical assistance in applying the models; develops and distributes guidance on air quality and source monitoring; establishes air quality indicators of progress, analyzes air pollution trends, and distributes information on progress in reaching air quality goals; conducts control strategy demonstrations, source monitoring, and ambient monitoring for OAQPS; develops new methods for ambient monitoring and modeling and develops and issues guidance and training materials to apply them; develops emission factors and provides technical guidance on emission inventories; and, conducts source testing ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 4 of 12 and develops new source test methods for use by Regional, state, and local clients. The Division analyzes air quality data for use in program evaluation and coordinates development and use of emission inventories in program evaluation. It promotes the integration and simplification of information and data management systems. It also serves as a source of technical expertise for OAQPS and provides technical support to Regional, state and local clients on source testing and methods. 1.2.3 Emission Standards Division The Emission Standards Division (ESD), under the supervision of its Director, is responsible for establishing emission standards (under Section 112 of the Clean Air Act) and managing federal programs for nationwide control of hazardous and criteria air pollutant emissions from stationary sources. The Division develops and implements emission standards for hazardous and criteria air pollutants, new source performance standards, control techniques guidelines, standards for hazardous wastes under the Resource Conservation and Recovery Act (RCRA), alternative control techniques documents, and guidance for implementing standards at the state and local level. It also conducts comprehensive studies of stationary source categories to determine the nature and magnitude of air pollution technology, control methods, operational and administrative procedures, and economic aspects of control. The Division is responsible for providing technical assistance to other Divisions in OAQPS, other offices in EPA, state and local agencies, small and large businesses, international organizations, and the public on effective emission control technology and associated costs. It develops overall plans, strategies, and policies addressing both regulatory programs for stationary sources of air toxics and criteria pollutants and innovative and streamlined approaches to regulatory development (including coordinated strategies for co-control of hazardous and criteria air pollutants). Finally, the Division establishes and maintains cooperative working relationships with Regional, state and local agencies, as well as selected stakeholders (industry, environmental groups, etc.), to facilitate effective development and implementation of regulations and guidance. 1.2.4 Air Quality Strategies and Standards Division The Air Quality Strategies and Standards Division (AQSSD), under the supervision of its Director, develops national and geographically focused strategies and programs for air quality management, based on assessments of health and ecological effects, exposure and risk, and economic impacts and benefits. Such assessments support the development of criteria pollutant ambient standards under Section 109 of the Clean Air Act (CAA), programs and emission standards under Sections 111 and 112, and the integration of various aspects of the criteria pollutant and hazardous air pollutant programs in coordination with other OAQPS divisions. More specifically, the Division manages the review and revision of NAAQS and develops attainment and maintenance strategies, policies, and implementation programs under Part D of CAA Title I, to guide the implementation efforts of the EPA Regional offices and state and local agencies. With respect to the hazardous air pollutant program, the Division conducts health and environmental effects evaluations in support of emission standards development, characterizes the nature and magnitude of the air toxics problem in the U.S., and develops air toxics control strategies. The Division serves as program source for expertise on health and ecological effects and the analysis of exposure and the risks associated with exposure to criteria and toxic air pollution. The Division also serves as program source for expertise on benefits assessments and on economic and regulatory impact analyses, including impacts on small entities and environmental justice, and on economic incentive programs. It conducts such analyses in support of OAQPS-wide standards, rules, and strategies, as well as in conjunction with Agency-wide assessments. Through its guidance, the Division develops and promotes the application of innovative, incentive-based regulatory strategies. The Division establishes and maintains cooperative working relationships with Regional, state and local agencies, as well as selected stakeholders (industry, environmental groups, etc.), to facilitate effective development and implementation of regulations, policies, and guidance. It also works closely with other OAQPS Divisions and EPA offices to ensure integration of rules, policies, and guidance to facilitate effective implementation. ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 5 of 12 1.2.5 Information Transfer & Program Integration Division The Information Transfer & Program Integration Division (ITPID), under the supervision of its Director, serves as the principal focus for the management and transfer of air pollution control information and for the integrated implementation of OAQPS programs, including operating permits. In carrying out these functions, the Division manages the design, development, maintenance, and evaluation of information systems, hardware, software, and other means of distributing key air pollution control information to government and non-government clients and the public at large. The Division also develops and delivers training courses and educational materials on various technical and management aspects of air pollution control to EPA, state, local, industry, and other relevant clients, and it assists other OAQPS and air program offices to conduct technical training workshops and to transfer critical guidance and information to program clients. In cooperation with other Divisions and programs, the Division promotes the integration and simplification of information delivery and data management systems. The Division manages and assures the integration of the national air quality permit programs, including operating permits, New Source Review, and Prevention of Significant Deterioration. In addition, the Division manages the implementation and integration of air toxics programs under section 112 with operating permit programs, to assure that the programs' requirements merge as smoothly as possible. The Division also assists the air programs which cut across Division, Office, and Agency lines, providing review of strategies and participating in the design of program improvements. 1.3 Key OAQPS Personnel 1.3.1 Management OAQPS Director The OAQPS Director has overall responsibility for managing OAQPS according to Agency policy and has final authority at the program office level. The direct responsibility for assuring data quality rests with line management. Ultimately, the Director is responsible for establishing QA policy and for resolving QA issues identified through the QA program. Major QA related responsibilities of the Director include: approving the budget and planning processes assuring that the OAQPS program develops and maintains a current and germane QMP and ensures adherence to the document by OAQPS staff and, where appropriate, other EPA offices and extramural cooperators assuring that QA Project Plans are developed for all environmental data operations in which OAQPS is the project lead and are submitted to the QA Manager(QAM) for review and approval prior to project initiation establishing policies to ensure that QA requirements are incorporated in all environmental data operations taking corrective action that may be required by the QA Manager's QA evaluation findings maintaining an active line of communication with the QA Manager The Director delegates the responsibility of QA development and implementation in accordance with Agency policy to the Division Directors. Oversight of the OAQPS QA program is delegated to the QA Manager in the Emissions, Monitoring, and Analysis Division (EMAD). ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 6 of 12 Joe Elkins OAQPS QA Manage John Seitz Director Emission, Monitoring, and Analysis Division Bill Hunt I Joe Elkins EMAD QA Representative Emission Standards Division Bruce Jordan I Doris Maxwell ESD QA Representative I MQAG QA Workgroup Joe Elkins Air Quality Strategies and Standards Division Sallv Sh 1 Maurice Walker AQSSD QA Representative I Information Transfer & Program Integration Division Bob Kellam I Mary Jane Clark ITPID QA Representative 7 Emissions I QA Workgroup I ^ Figure 1.2 OAQPS QA Team organizational structure EMAD Division Director The EMAD Division Director serves a dual role as Director of an OAQPS Division and as the one with oversight of the QA Program and the QAM. Major responsibilities include: ensuring that Division staff follow the OAQPS QMP supporting the team members with required resources meeting regularly with the QAM to provide feedback and guidance approving recommendations advocating the team cause and working to overcome barriers 1.3.2 Quality Assurance The OAQPS QA Team is responsible for ensuring that OAQPS management and staff understand and adhere to the OAQPS QMP. Figure 1.2 represents the organization structure of the OAQPS QA Team. Quality Assurance Manager The QAM is the delegated manager of the OAQPS QA Program. He has direct access to the OAQPS Director on all matters pertaining to Quality Assurance. The main responsibility of the QAM is QA oversight; ensuring that all personnel understand the OAQPS QMP and understand their QA/QC responsibilities. The QAM provides technical support and reviews, and approves QA products. Responsibilities include: interpreting Agency QA policy and developing the QA policy for OAQPS in accordance with Agency QA policies and direction from management developing a QMP and revising it as necessary developing a QA Annual Report and Work Plan for the OAQPS Director and the Agency's Quality Assurance Division (QAD) of the National Center for Extramural Research and Quality Assurance reviewing acquisition packages (contracts, grants, cooperative agreements, inter-agency agreements) to determine the necessary QA requirements and signing to certify that the review took place ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 7 of 12 developing QA budgets assisting staff scientists and project managers in developing QA documentation and in providing answers to technical questions ensuring that all personnel involved in environmental data collection have access to any training or QA information needed to be knowledgeable in QA requirements, protocols, and technology ensuring that environmental data operations are covered by appropriate QA planning documentation (e.g., QA project plans and data quality objectives) tracking the QA/QC status of all programs assisting in solving QA-related problems at the lowest possible organizational level recommending required management-level corrective actions serving as the program's liaison with QAD The QAM has the authority to carry out these responsibilities and to bring to the attention of the Director of OAQPS any issues associated with these responsibilities. Quality Assurance Division Representatives The QA Division Representatives are the main points of contact within each of the four OAQPS Divisions. The QA Representative and the OAQPS QA Manager make up the QA Committee. Their responsibilities include: implementing the OAQPS QMP within the Division acting as a conduit for QA information to Division staff representing the Division's interests on the QA Team assisting the QAM in developing QA policies and procedures coordinating the Division's input to the QMP and the Quality Assurance Annual Report and Work Plan(QAARWP) Each QA Division Representative has the authority to carry out these responsibilities and to bring to the attention of his or her respective Division Director any issues related to these responsibilities. Deputy Quality Assurance Officers The Deputy QA Officers are the official staff QA contacts appointed by the OAQPS Director. Their responsibilities include: remaining current on QA developments reviewing and approving QA forms for contracts and grants reviewing and approving QA project plans and QMPs for projects involving EDO serving on the QA Team The DQAOs in ESD, AQSSD, and ITPID report directly to their respective Division Director. In EMAD the DQAOs report directly to the Group Leaders responsible for primary data collection activities or analysis of that data. 1.3.3 Quality Assurance Team Team Mission The Office of Air Quality Planning and Standards Quality Assurance Team is dedicated to ensuring that environmental data operations are of a quality that meets or exceeds requirements for informed ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 8 of 12 environmental decision- making. The Team remain committed to providing the OAQPS Director information, guidance, and expertise to ensure decisions are made to protect the public and the environment. The Team recognizes that high quality products will be achieved through effective communication, training, cooperation, and a desire to produce the best results possible. The Quality Assurance Team includes the QAM and the QA Division Representatives (QA Committee), the Deputy QA Officers, and other OAQPS staff members with specific QA responsibilities. QA Team meetings are an opportunity for all QA staff to meet and discuss issues across the divisions. The responsibilities of the QA Team include: developing the QMP and the QAARWP developing and reviewing QA policies and procedures implementing the QMP identifying evolving QA issues promoting QA within OAQPS and with cooperating organizations providing QA support to other staff and external organizations In addition to having the responsibilities noted above, the QA Team members carry out many varied support functions within OAQPS. They support environmental data collection and analysis activities, both internal and external to EPA, including Regional Offices, state and local agencies, and businesses. Individual responsibilities are detailed in the employee's performance standards. 1.3.4 OAQPS Staff Project Managers/Work Assignment Managers OAQPS Project Managers and Work Assignment Managers are responsible for the performance and coordination of specific projects and are management 's and the QAM's principal contacts regarding these projects. Project Officers and Work Assignment Managers are responsible for all data collection activities, and they determine the QA criteria to be applied, based on the intended use of the data. QA criteria for projects are communicated through the development of data quality objectives (DQOs) and QA project plans. Responsibilities include: developing, or assisting in the development of DQOs developing, or ensuring the development of the QA Project Plan through negotiation with contractors, appropriate QA representatives, and other technical personnel when needed, then having this plan submitted, reviewed, and approved prior to project initiation ensuring the implementation of QA project plans ensuring that standard operating procedures (SOPs) for each data collection operation are reviewed and approved reviewing project QA/QC outputs developing, or ensuring the development of, QA Reports OAQPS Group Leaders In addition to the specific responsibilities listed above, OAQPS Group Leaders are responsible for ensuring that subordinate personnel are trained and follow the policies of the QMP. Each leader is responsible for the quality of the environmental data collected in his or her program. ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 9 of 12 OAQPS Personnel In addition to the specific responsibilities listed above, all OAQPS personnel are responsible for their own work. They are expected to be familiar with the policies of the QMP and implement them in any activity. 1.4 Technical Activities Supported by the Quality System In order to support its mission, OAQPS funds and implements many environmental data operations (EDOs), and these must be supported by the quality system. One major EDO are the programs related to air quality which is represented by figure 1.3. OAQPS serves as a major focal point for activities in fulfillment of the Clean Air Act. It utilizes the services of contractors, grantees, other Federal agencies, and EPA Regional support to generate necessary technical data and reports of findings. EDOs include: Assessment and mitigation of the effects of air pollution, through ambient and source monitoring of the air Development of environmental indicators and new methods for ambient and source monitoring Developing and distributing guidelines for air quality models Developing emission factors and standards and providing technical guidance on emission inventories Developing New Source Performance Standards Conducting comprehensive studies of stationary source categories to determine the nature and magnitude of air pollution emissions Source testing Evaluation and demonstration of remedial actions and control strategies for air pollution Assessments of health and ecological effects, exposure and risk, extent and trends of air pollution contamination Reviewing and revising the National Ambient Air Quality Standards Managing a large air pollution data base. Measurements producing data would include: Sampling, and analysis of a host of air pollution constituents, from both continuous automated instruments and manual instruments Mapping and landscape characterization techniques through various remote sensing platforms Meteorological and physical parameters Details on the data collection activities of each OAQPS Group (shown in Figure 1.1), can be found in Appendix A. OAQPS EDOs include all phases of the data collection activities including field sampling, laboratory analysis, data management, modeling, assessment, and reporting. In general, OAQPS EDOs focus on the collection of data from two major program areas: 1) ambient air data and 2) emission data. These data, along with secondary data, are interpreted and reported and are also used in modeling efforts to perform Figure 1.3 Air quality management process ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 10 of 12 additional interpretation and assessment. Figure 1.3 provides a generic representation of the role of collected data in the air quality management process. The major OAQPS data collection activities are discussed below. 1.4.1 Ambient Air Data The Monitoring and Quality Assurance Group (MQAG) of the Emissions, Monitoring, and Analysis Division (EMAD) is responsible for the Ambient Air Monitoring Program which collects air quality samples generally for one or more of the following purposes: To judge compliance with and/or progress made towards meeting ambient air quality standards To activate emergency control procedures that prevent or alleviate air pollution episodes To observe pollution trends throughout a region, including non-urban areas To provide a data base for research evaluation of effects: urban, land-use, and transportation planning; development and evaluation of abatement strategies; and development and validation of diffusion models. To evaluate potential alternative forms of the National Ambient Air Quality Standards With the end use of the air quality samples as a prime consideration, a monitoring network designed to meet one of four basic monitoring objectives: 1. to determine highest concentrations expected to occur in the area covered by the network 2. to determine representative concentrations in areas of high population density 3. to determine the impact on ambient pollution levels of significant sources or source categories 4. to determine general background concentration levels that the monitoring network will collect, which must be representative of the spatial area being studied The EPA's ambient air quality monitoring program is carried out by state and local agencies and consists of three different categories of monitoring stations that measure the criteria pollutants: particulate matter, sulfur dioxide, carbon monoxide, nitrogen dioxide, ozone, and lead. Additionally, a fourth category of monitoring station which measures ozone precursors (approximately 60 volatile hydrocarbons and carbonyl), has been required by the 1990 Amendments to the CAA. The following information describes these four networks. State and Local Air Monitoring Stations (SLAMS) - The SLAMS consist of a network of ~ 4,000 monitoring stations whose size and distribution are largely determined by the needs of state and local air pollution control agencies to meet their respective State Implementaion Plan (SIP) requirements. National Air Monitoring Stations (NAMS) - The NAMS (-1,080 stations) are a subset of the SLAMS network, with emphasis being given to urban and multi-source areas. In effect, they are key sites under SLAMS, reflecting areas of maximum concentrations and high population density. Special Purpose Monitoring Stations (SPMS) - Special Purpose Monitoring Stations provide for special studies needed by the state and local agencies to support SIPs and other air program activities. The SPMS are not permanently established and, thus, can be adjusted easily to accommodate changing needs and priorities. The SPMS are used to supplement the fixed monitoring networks as circumstances require and resources permit. If data from SPMS are used for SIP purposes, they must meet all QA and methodology requirements for SLAMS monitoring. Photochemical Assessment Monitoring Stations (PAMS) - A PAMS network is required in each ozone nonattainment area that is designated serious, severe, or extreme. The required networks will have from two to five sites, depending on the population of the area. There is a 5-year phase-in period of one site per year, starting in 1994. The ultimate PAMS network could exceed 90 sites at the end of the phase-in period. ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 11 of 12 In-house/External Air Monitoring Projects- Internally, OAQPS does not physically implement any EDOs (e.g., field sampling or analysis) for ambient air monitoring data. All data collection activites are externally funded to Regions, state and local organizations. 1.4.2 Emissions Data Collecting emissions data involves a variety of OAQPS programs, including source characterization. Source Characterization The Source Characterization Groups (SCGs) of the Emissions, Monitoring, and Analysis Division (EMAD) are responsible for obtaining source emission data and for assisting OAQPS in data-gathering activities. These activities fall into four categories: 1. Category I - EPA field testing to support the development of New Source Performance Standards, National Emission Standards for Hazardous Air Pollutants, and emission standards under SIPs. 2. Category II - EPA evaluation of state, local, and industry generated emission measurement data and information used for purposes of supporting rule, method, or monitoring system development by the Agency. Such activity encompasses the determination of data quality in relation to anticipated defense of supporting program data quality objectives. 3. Category III - Field testing concerning performance test methods and source monitoring procedures, evaluate formal requests to allow the use of alternative methods, or investigating the feasibility of new or modified performance test methods or source monitoring procedures. 4. Category IV - Field testing for purposes not included in Category I or II, such as emission screening studies to assess regulatory needs, testing to generate emission factors, or pollutant assessment projects. In-house Source Emission Measurement Projects In-house data evaluation and quality assessments and testing are usually limited to projects in Category II and III. The SCGs project leader of this work is responsible for developing a project plan and data quality objectives, in concert with the requesting supporting program Project Leader, and for obtaining Group Leader approval for the work to be done. This project plan must include a QAPP to ensure that defensible data of known quality and integrity are generated. The SCGs project leader is also responsible for preparing or referencing SOPs, when applicable. The Group's DQAO reviews and approves the QAPP and also seeks concurrence from the DQAO of the requesting support program Group. Extramural Source Emission Measurement Projects Almost all SCGs field test projects in Categories I and IV are conducted through cost-plus-award-fee level-of-effort contracts or through agreement with the private sector. Those agreements with the private sector are where OAQPS agrees to accept and consider for use environmental data collected, documented and paid for voluntarily by a private sector party. The private sector parties are requested to submit a QAPP. OAQPS is under no obligation to use this data unless it is determined to be adequate and sufficient for their intended use. ------- OAQPS Section 1 Revision: 01 Date: 06/21/96 Page 12 of 12 For SCGs contracts, technical proposals must include detailed discussions of the company's QA program and organization of QA activities. These discussions include development of QAPPs, responsibilities of the contractor's QAO, periodic QA activity updates, and corrective action plans for adverse QA findings. Only contractors with high quality QA programs are selected for final award; and consideration is enhanced where the contractor's QAO can function independently of the testing program. For specific work assignments, individual work plans and test reports are reviewed, emission test procedures are observed, as resources permit, and performance evaluation audit samples are administered by the appropriate SCG WAM to ensure high quality emission data. Examples of high quality work plans and test reports are available to the WAM as minimum acceptable criteria. 1.4.3 Modeling Modeling is defined as the development of a mathematical or physical representation of a system or theory that accounts for all or some of its known processes. Models are often used to test the effect of changes of components on the overall environmental system or theory. OAQPS provides leadership and direction on a full range of dispersion models and other mathematical techniques, to assess and control emission sources and to support policy/regulatory decisions in OAQPS. OAQPS also develops emission factors that go through a type of modeling process to generate these factors. 1.4.4 Secondary Data (Secondary Use of Data) OAQPS uses secondary data in many of its assessments and models. Secondary data can be considered data that are utilized for a purpose other than that for which they were initially collected. Secondary data are sometimes used by OAQPS in the following standards development processes : Maximum Achievable Control Technologies (MACT) Best Achievable Control Technologies (BACT) Reasonably Available Control Technology (RACT) Lowest Achievable Emission Rate (LAER) ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 1 of 6 2. OAQPS Quality System and Description A quality system is defined as a structured and documented management system describing the policies, objectives principals, organizational authority, responsibilities, accountability, and implementation of an organization for ensuring quality in its work processes, products (items), and services. The quality system provides the framework for planning, implementing, and assessing work performed by the organization and for carrying out required QA and QC. This section will describe the quality system applications used by the organization to implement effective quality assurance activities for its environmental data operations (EDOs). 2.1 OA QPS QA Application In order to meets its stated mission using environmental data, OAQPS must implement a QA program that assures that the data can be used for its intended purpose. The following elements will assist in the assurance of data quality and will be described. QA management plans Management systems reviews Data quality objectives process QA project plans Standard operating procedures Data quality assessments. Various reviews to determine the successful application of QA in OAQPS will be discussed in Section 9 and 10. 2.1.1 Quality Management Plans EPA policy requires that all Agency organizational units document their QA program in an approved Quality Management Plan (QMP). The document describes the quality system in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing all activities conducted. The QMP is developed for use by all OAQPS staff, as detailed in Section 1. The QMP will reside on the RTP wide web for easy access to all OAQPS staff. A hardcopy will also be filed with the OAQPS Document Control Manager (see section 5). Approval for the QMP will include the OAQPS QA Manager, Division Directors, the OAQPS Director. It is then submitted for approval by the Director, National Center for Environmental Research and Quality Assurance, under delegation from the Assistant Administrator for Research and Development (AA/ORD), based on an affirmative recommendation by the Director, QAD. This approval is valid for up to five years, pending changes to the organization's quality system during the interim. The QMP will be reviewed every August by the QA Committee to determine if the information remains relevant to the Office. A briefing of the findings will be provided to senior management. If changes are required, they will be made by October 1 of each year. Any change will include a change in the revision number of the section and the date (possibly page numbers). The change will also be reflected in the table of contents. A description of the changes will be sent out via E-mail to all OAQPS staff, appropriately archived, and will be included in the QAARWP. A hard copy of revised sections and the table of contents will be sent to QAD in order to keep their copy of the OAQPS QMP current. Every 5 years, based upon the original approval date, the QMP will undergo a thorough review, in its entirety, and go through the approval cycle. Since the QMP undergoes yearly reviews, this would simply be another yearly review with the addition of approval signatures. ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 2 of 6 2.1.2 Management Systems Reviews A management systems review (MSR) is a qualitative assessment of data collection operations and/or organization(s) to establish whether the prevailing quality management structure, policies, practices, and procedures are adequate for ensuring that the quality of data needed are obtained. It is used to determine the effectiveness of, and adherence to the QA program and the adequacy of resources and personnel provided to achieve and ensure quality in all activities. Details of the MSR process can be found in section 9. QAD plans to implement independent MSRs of every EPA organization once every three years. An internal MSR of the OAQPS QA program will be conducted at the same regularity, but will be staggered to the QAD MSR in order to provide some type of MSR every 1.5 years. A more frequent internal review cycle may be required if serious deficiencies exist. Results of the MSR will be documented and archived, as specified in the OAQPS Document Management Plan (see Section 5), and included in the appropriate Quality Assurance Annual Report and Work Plan (QAARWP). 2.1.3 Data Quality Objectives (DQOs) The data quality objectives (DQOs) are qualitative and quantitative statements derived from the DQO process that clarify project technical and quality objectives, define the appropriate type of data, and specify tolerable levels of potential decision errors that will be used as the basis for establishing the quality and quantity of data needed to support decisions. When a formal DQO document is required, it is the responsibility of the OAQPS Project Officer or Work Assignment Manager (WAM) to define this allowable uncertainty and develop DQOs with the principal investigators and cooperators. A formal DQO document is required for all projects that require Category I QA project plans (see section 2.1.4). DQOs for the other 3 categories of QA project plans can be included in the QA project plan. The DQO process is used to facilitate the planning of data collection activities. It asks the data user to focus their planning efforts by specifying the use of the data (the decision), the decision criteria, and the probability they can accept of making an incorrect decision based on the data. The DQO process: Establishes a common language to be shared by decision makers, technical personnel, and statisticians in their discussion of program objectives and data quality. Provides a mechanism to pare down a multitude of objectives into major critical questions. Facilitates the development of clear statements of program objectives and constraints which will optimize data collection plans. Provides a logical structure within which an iterative process of guidance, design, and feedback may be accomplished efficiently and cost effectively. DQO development should be a normal part of the project planning process and must be accomplished based on cost-effectiveness and realistic capabilities of the measurement process. To facilitate this determination, the Quality Assurance Division (QAD) of EPA (formally known as QAMS) developed the DQO Process in 1984. Guidance for the Data Quality Objectives Process, which is issued by QAD, can be obtained from each division QA representative. Training software on the DQO process can also be acquired from QAD. The Decision Error Feasibility Trials (DEFT) software was developed to allow a decision maker or member of the DQO planning team to quickly generate cost information about several simple sampling designs based on the DQO constraints. This software is available through the OAQPS QA Manager or by contacting QAD. The DQO process assists the user in defining the purpose for an environmental data operation and sets ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 3 of 6 the framework for the design, implementation, and QA of the project. Once DQOs are defined, a QA program can be developed. By using the DQO Process to plan EDOs, EPA can improve the effectiveness, efficiency, and defensibility of decisions in an effective manner. DQOs are being utilized and developed for both the Ambient Air and Emission Programs 2.1.4 QA Project Plans The QAPP is a formal document describing in comprehensive detail the necessary QA, QC, and other technical activities that must be implemented to ensure that the results of work performed will satisfy the stated performance criteria (DQOs). The information in this section applies equally to in-house and extramural QAPPs. QAPPs are not required for projects that do not include EDOs. The quality assurance policy of the EPA requires every EDO to have written and approved quality assurance project plans (QAPPs) prior to the start of the EDO. It is the responsibility of the Program Manager/WAM to adhere to this policy. If the Program Manager/WAM proceeds without an approved QAPP, he/she is fully aware of the risks and assumes all responsibility. The Program Manager/WAM also bears the responsibility of providing copies of the approved QAPP to each individual who has a major responsibility in the EDO and explaining the elements of the QAPP to these individuals. QAPP Review and Approval QAPPs are prepared, reviewed and approved in accordance with EPA QA/R5, EPA Requirements for Quality Assurance Project Plans for Environmental Data Operations. Copies of this document are available from the Division QA Representative. This document identifies and defines the elements that must be addressed in all formal QAPPs. The following subsection presents criteria for the determination of the type of QAPPs necessary for OAQPS EDOs. Review of the QAPP must include QAPP preparer, Program Manager/WAM, and either the QAM or DQAO. It is recommended that the document be initially reviewed by the Program Manager/WAM before submission to the QAM or DQAO. It is also recommended that these plans be reviewed by a statistician. The QAM or DQAO will review each QAPP for the required elements and the soundness of the QA/QC. The QAM or DQAO will attempt to review QAPPs within 15 working days of submission. The QAM or DQAO will provide written comments on each element which will be accompanied by a completed QAPP Review Form (Appendix C). The review form is a summary that alerts the Program Manager/WAM as to whether or not QA requirements have been adequately achieved. Through the QAPP Review Form, the QAM/DQAO will determine whether the QAPP is worthy of approval, and if not, will identify those elements requiring revision. If the QAPP requires revision it will be sent back to the author. The revisions which may be included in the QAPP or as an addendum must be reviewed and approved by the QAM/DQOA. All QAM or DQAO QAPP reviews are secured in a file by the contract ID number in the respective QAM or DQAO office. QAPP Revision As mentioned in the paragraph above, any revisions required to the original QAPP can be included in a second or subsequent revision or an addendum. However, sometimes the scope of a project can change which may have the potential to affect the quality of the data. If these changes affect the collection of environmental data, an addendum to the approved QAPP must be submitted that describes the changes and the appropriate QA/QC techniques necessary to meet the DQOs. The Contract Manager/WAM must approve the changes. ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 4 of 6 QAPP Archive QAPPs should be filed with the OAQPS Document Control Officer (DCO) who will identify the document with a unique document control number (see section 5). Tracking of QAPPs will be accomplished by the OAQPS QAM or DQAO. At present this will be accomplished manually, but an automated system is planned. All original copies of the QAPPs and any subsequent revisions will be secured by the DCO. The QAM or DQAO will maintain a copy for the QA files. If possible, a disk copy of QAPPs should also be acquired. Categories of QA Project Plans OAQPS will utilize a four-tiered project category approach to its QA Program in order to effectively focus QA, as discussed in EPA QA/R5, EPA Requirements for Quality Assurance Project Plans for Environmental Data Operations. Each QAM or DQAO will be provided a copy of these documents. A number of reference copies will also be available through the QAM. This approach was originally developed by the U.S. EPA, Air and Energy Engineering Research Laboratory (AEERL) and published by the EPA Risk Reduction Engineering Laboratory, Cincinnati, Ohio (EPA/600/9-89/087). Category I involves the most stringent QA approach, whereas Category IV is the least stringent. The following definition of the categories are quoted from the document listed above: Category I Projects Projects include environmental data operations that directly support rulemaking, enforcement, regulatory, or policy decisions. They also include research projects of significant national interest, such as those typically monitored by the Administrator. Category I projects require the most detailed and rigorous QA and QC for legal and scientific defensibility. Category I projects are typically stand-alone; that is, the results from such projects are sufficient to make the needed decision without input from other projects. Category II Projects Projects include environmental data operations that complement other projects in support o f rulemaking regulatory, or policy decisions. Such projects are of sufficient scope and substance that their re suits could be combined with those from other projects of similar scope to provid e necessary information fo r decisions. Category II projects may also include certain high visibility projects as defined by EPA management. Category III Projects Projects include environmental data operations performed as interim steps in a larger group o f operations. Such projects include those producing results that are used to evaluate and selec t options for interim decisions or to perform feasibility studies or preliminary assessments o f unexplored areas for possible future work. Category IV Projects Projects involving environmental data operations to study basic phenomena or issues, includin g proof of concepts, screening for particular analytical species, etc. Such projects generally do not require extensive detailed QA/QC activities and documentation. The number of elements required for each category is reduced as one proceeds from category I to IV as ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 5 of 6 illustrated in Table 2-1. EDOs that meet the requirements for a formal DQO process (Sub-section 2.1.3) will be required to develop category I QAPPs. Program Managers/WAMs, with consultation from the QAM or DQAO, will be required to designate the category of the QAPP. The statement of the category will be placed on the QAPP signature and approval page which will include signatures of the QAPP preparer, Program Manager/WAM, and the QAM or DQAO. Others personnel, depending upon the nature (intramural or extramural) of the QAPP, may need to sign the approval page. Table 2-1 QAPP Elements Applicable to Various Categories QAPP Element Category Applicability A1 Title and Approval Sheet I, II, HI, IV A2 Table of Contents I, II, HI A3 Distribution List I, II, A4 Project/Task Organization I, II, HI A5 Problem Definition/Background I, II, HI A6 Project/Task Description I, II, HI, IV A7 Quality Objectives and Criteria for Measurement Data I, II, III, IV A9 Special Training Requirements/Certification I A10 Documentation and Records I, II, III B1 Sample Process Design I, II, III, IV B2 Sampling Methods Requirements I, II, III, B3 Sample Handling and Custody Requirements I, II, III B4 Analytical Methods Requirements I, II, III, IV B5 Quality Control Requirements I, II, III, IV B7 Instrument Calibration and Frequency I, II, III B8 Inspection/Acceptance Requirements for Supplies and 1,11 Consumables B9 Data Acquisition Requirements I, II, III BIO Data Management 1,11 CI Assessments and Response Actions I, II, III C2 Reports to Management I, II, III D1 Data Review, Validation, and Verification I, II, III Requirements 1,11 D2 Validation and Verification Methods I, II, III D3 Reconciliation and User Requirements In-house Quality Assurance Project Plans. All EDOs accomplished by OAQPS staff (e.g any federal or private employee retained for OAQPS services and located at the OAQPS offices) must be covered by an approved QAPP prior to the start of the EDO. Mandatory approval signatures include the Group Leader, Principal Investigator, and the OAQPS QAM or DQAO. Extramural Quality Assurance Project Plans The Grant and Federal Assistance Regulations 40 CFR 1 parts 30.53 and 31.45 document the QA requirements when the project entails EDOs. One of the four categories of QAPPs, as determined by the Project ------- OAQPS Section 2 Revision 01 Date: 5/96 Page 6 of 6 Manager/WAM, will be required. Mandatory approval signatures include the QAPP preparer, Program Manager/WAM, and the OAQPS QAM or DQAO. 2.1.5 Standard Operating Procedures (SOPs) Standard operating procedures (SOPs) are written documents that detail the method for an operation, analysis, or action with thoroughly prescribed techniques and steps. It is officially approved as the method for performing certain routine or repetitive tasks. SOPs are protocols for all routine activities, especially those that are involved in the environmental data operations, which generally involve repetitious operations performed in a consistent manner. SOPs should ensure consistent conformance with organizational practices, serve as training aids, provide ready reference and documentation of proper procedures, reduce work effort, reduce error occurrences in data, and improve data comparability, credibility, and defensibility. They should be sufficiently clear and written in a step-by-step format to be readily understood by a person knowledgeable in the general concept of the procedure. Elements to include in OAQPS SOPs will follow the QAD document entitled Guidance for the Preparation of Standard Operating Procedures (SOPs) EPA QA/G-6. Copies of this document are available at each Division QA Representatives office, as well as through the QAD office. SOPs should be written by individuals performing the procedures that are being standardized. SOPs for data collection methods must be included in QAPPs either by reference or by inclusion of the actual method. If a method is referenced, it must be stated that the method is followed exactly or an addendum that explains changes to the method must be included in the QAPP. If a modified method will be used for an extended period of time, the method must be revised to include the changes to appropriate sections. In general, approval of SOPs occur during the approval of the QAPP. Individuals with appropriate training and experience with the particular SOPs in the QAPP need to review the SOPs. Internal OAQPS SOPs must be approved by the supervisor of the personnel responsible for writing the document. An SOP cannot be revised during an EDO without the consent of the Contract Manager/WAM. If the modification is accepted, it must be documented in a letter to the Contract Manager/WAM and reported in the next progress report. 2.1.6 Data Quality Assessments (DQAs) Data Quality Assessments (DQAs) provide important information that allow the decision maker to determine whether the data produced from an EDO support their intended use (the DQOs). QAD has recently developed a document entitled Guidance for Data Quality Assessment Practical Methods for Data Analysis EPA QA/G-9 (QA96 version) which can be used to assist in the DQA process. DQA is a statistically-based, quantitative evaluation of the extent to which the quality of the results from environmental data operations supports the decision made through the DQO process mentioned above. DQA requires a joint effort between the decision maker and a statistician. The decision makers (generally the Project Officer or WAM) contributions consist of an inspection of data for scientific anomalies and transcription errors, assessment of the effect of QA/QC deviations, and professional contextual judgement. The statistician, on the other hand, is responsible for the graphical display of data and trends, statistical analysis required by the DQO including assessments of data uncertainties, investigation of assumption violations, identification of potential outliers, and direction for data improvement. ------- OAQPS Section: 3 Revision: 01 Date: 6/96 Page:l of 3 3. Personnel Qualifications and Training 3.1 Train ing Policy The staff members of OAQPS are expected to have met the educational, work experience, and training requirements for their positions, as outlined by the Office of Personnel Management in their position descriptions. OAQPS requires appropriate training for its employees, supporting contractors, and grantees. Such training may consist of classroom lectures, workshops, teleconferences, and on-the-job training. Courses may be mandatory, such as safety, Confidential Business Information (CBI), and contract and grant management; or nonmandatory, such as leadership, teamwork, ambient and stationary source data collection, and the development of quality assurance project plans (QAPP) and data quality objectives for data collection and analyses. Training needs are identified each year by OAQPS employees in their Individual Development Plans (IDP) as part of their self- evaluation process. The IDP is meant to help ensure that the staff members remain current in their technical fields, that they have opportunities for growth, and that they are able to meet the challenges of changing agency vision and goals. It is the responsibility of OAQPS senior management to anticipate, identify, and communicate effectively, to both OAR management and OAQPS staff, any changes needed in both the OAR and OAQPS vision and goals. It is also the OAQPS senior management responsibility to establish a systematic measurement system to identify and comply with mandatory (statutory and regulatory) training requirements, to assess success of current nonmandatory training, and to identify opportunities for future cost-effective improvements. It is the group leader's responsibility to encourage OAQPS staff to gain nonmandatory training needed to effectively accomplish current and future OAQPS goals. The IDP review process is one formal mechanism for accomplishing this goal and for assuring compliance with mandatory training requirements. 3.2 Mandatory Training OAQPS has established a systematic procedure for safeguarding CBI, which is detailed in a written security manual. Staff who may have access to CBI are identified by their group leaders and must receive specific training and pass a certification course. The security manual details the educational and training requirements. The current version is dated June 1995 and it is updated periodically. The Director of OAQPS or his/her designee is the responsible official. This authority to direct and administer the CBI program at OAQPS has been delegated to the Emission Standards Division Director. All OAQPS field and laboratory staff are required to complete an initial safety program and must periodically take refresher courses. The OAQPS Director is the responsible official. To assist him in administering these requirements, he has designated a Project Manager for Safety, Health, and Environmental Management at the staff level. Each Division Director (DD) has been further delegated responsibility to a staff person for assuring staff compliance with mandatory safety training. OAQPS has established a procedure for assuring that contracts and grants are effectively managed. One aspect of effective management is assuring that staff has mandatory training. The OAQPS Director is the responsible official. Each DD has been further delegated responsibility for assuring staff compliance with training requirements. POs and WAMs must take certification courses and refresher courses in order to serve in an official capacity. The Office of Administration and Resources Management's Contracts Management Division and Grants Administration Division have further oversight responsibilities of the POs' and WAMs' work. Further information on any mandatory training can be obtained from the DDs' designees. ------- OAQPS Section: 3 Revision: 01 Date: 6/96 Page:2 of 3 3.3 Recommended Training Formal QA/QC training is offered through the following organizations: Air Pollution Training Institute (APTI) Air & Waste Management Association (AWMA) American Society for Quality Control (ASQC) EPA Institute EPA Quality Assurance Division (QAD) EPA Regional Offices In addition, OAQPS uses contractors and academic institutions to develop and provide training for data collection activities that support regulatory efforts throughout OAQPS, as well as the states and Regions. The OAQPS QA Program maintains a list of available courses. Table 3-1 provides a suggested sequence of core QA-related ambient air monitoring courses for OAQPS ambient air monitoring staff, emissions monitoring staff, and QA managers (marked by asterisk). The suggested course sequences assume little or no experience in QA/QC or air monitoring. Persons having experience in the subject matter described in the courses would select courses according to their appropriate experience level. Courses not included in the core sequence would be selected according to individual responsibilities, preferences, and available resources. Table 3-1. Suggested Sequence of Core QA-related Ambient Air and Emission Monitoring Training Courses for OAQPS QA or Ambient Air Monitoring Contacts and QA Managers Sequence Course Title (SI = self instructional) Source Ambient Emissions 1* 1* Air Pollution Control Orientation Course (Revised), SI:422 APTI 2* 2* Principles and Practices of Air Pollution Control, 452 APTI 3* 3* Orientation to Quality Assurance Management QAD 4* Introduction to Ambient Air Monitoring (Under Revision), SI:434 APTI 5* General Quality Assurance Considerations for Ambient Air Monitoring (Under Revision), SI:471 APTI 6* Quality Assurance for Air Pollution Measurement Systems (Under Revision), 470 APTI 4* Quality Assurance for Source Emission Measurements, SI:414 APTI 5 Transmissometer Systems - Operation and Maintenance, SI:476A APTI 6 Continuous Emission Monitoring System - Operation and Maintenance of Gas Monitors, SI:476B APTI 7 Measuring the Emission of Organic Compounds to the Atmosphere, SI:483 APTI ------- OAQPS Section: 3 Revision: 01 Date: 6/96 Page:3 of 3 Sequence Course Title (SI = self instructional) Source Ambient Emissions 7* 8* Data Quality Objectives Workshop QAD 8* 9* Quality Assurance Project Plan QAD 9 Atmospheric Sampling (Under Revision), 435 APTI 10 Continuous Emission Monitoring for the 90s AWMA 10 Analytical Methods for Air Quality Standards, 464 APTI 11 11 Chain-of-Custody Procedures for Samples and Data, SI:443 APTI * * Data Quality Assessment QAD * * Management Systems Review QAD * * Beginning Environmental Statistical Techniques (Revised), SI:473A APTI * * Introduction to Environmental Statistics, SI:473B APTI * * Quality Audits for Improved Performance AWMA * * Statistics for Effective Decision Making ASQC * Courses recommended for QA Managers As noted above, the courses listed in this section are currently recommended and are not mandatory. The OAQPS QA Program is considering whether some of these courses should be required for QA officers and staff that are involved with environmental data operations. One major consideration is making sure that any required courses are readily available by being frequently taught, available on video, or available as self-instructional courses. OAQPS plans to use EPA QA/G-7 Guidance for Determining Quality Training Requirements for Environmental Data Operations to develop training requirements for those involved in assessment and response activities, including management systems reviews and audits. Until that guidance is available the OAQPS Supervisors must use their judgement to ensure that each time they assign someone to an assessment and response activity that the individual assigned has: 1) the correct technical expertise; and 2) the appropriate audit and/or management systems review training. ------- OAQPS Section 4 Revision: 01 Date: 5/96 Page 1 of 3 4. Procurement of Items and Services OAQPS must ensure that the items and services it acquires are procured within EPA regulations, are delivered in a timely fashion, and are within the required specifications. The following sections will provide general guidance on OAQPS procurement procedures. 4.1 Procurement of Items The Contracts Management Division (CMD) follows the guidelines developed in the Federal Acquisition Regulations (FAR) section 13, which establishes government-wide policies and procedures governing the acquisition process. Two EPA documents: EPA 1900-Contract Management Manual and the EPA Acquisition Regulation Manual (EPAAR), have been developed to supplement the FAR. EPA attempts to purchase through FAR- mandatory sources (i.e., GSA). Therefore, items on the FAR source list that meet the minimum specifications on the procurement request (EPA Form 1900-8) must be purchased through a FAR source. Procurements of computer hardware and software are subject to somewhat different regulations and should be coordinated with your PC site coordinator. In EPA, only contracting officers (COs) are authorized to procure items and services, unless it is an imprest fund transaction approved by the CO prior to the originators purchase of the item. The Federal Government is not bound by any commitments made by other than authorized personnel. Requests for purchases begin at the planning stages of any OAQPS project, and funds must be identified in the project scope of work for such purchases. All items should be identified and specifications that meet the government's minimum needs should be detailed. These specifications will be referred to during the procurement process and will assure that the OAQPS requestor receives the proper item and reduces the chances of purchase delays or incorrect purchases because of inadequate product specifications. All procurements are documented using EPA Form 1900-8 (PR). Instructions are included with the form. A purchasing agent will inform the originator of the item that most closely matches his or her request that is available from the FAR mandatory sources. A purchasing agent may "compete" a purchase on "brand name or equal" specifications. Manufacturer names and model numbers help make a description complete. This does not mean that the brand name will be ordered. If the item available from the mandatory source does not meet specifications, and no substitute is adequate, a purchasing agent will help the originator process a Waiver Request. However, if the items' total price are less than $2,500, and the type of item is not available through mandatory sources, Purchasing may buy from the suggested source. Always include a suggested source in block 13 on EPA Form 1900-8. PRs will be reviewed for completeness and accuracy by the appropriate approving official. The PR will then be forwarded to other required approvals, such as from the Senior Information Resource Management Official (SIRMO) for automated data processing (ADP) purchases. Funds are certified as available by a funds certifying officer, who assigns a document control number (DCN). The PR is forwarded to the Property Management Officer for signature and then to CMD. The sign-off chain may take up to 2 weeks from the time the PR is typed until it arrives in CMD. If the supply/service is required sooner, the PR may be "walked through" the sign-off chain and must contain a priority justification. Government bank card purchases do not require CO approval, since the bank card holder is delegated this authority. Unauthorized purchases will not be reimbursed without submittal of a "Ratification of Unauthorized Commitment Form". This process is very time consuming and costly and requires approval of CMD. ------- OAQPS Section 4 Revision: 01 Date: 5/96 Page 2 of 3 4.2 Procurement of Services Two types of mechanisms are primarily used to procure services, contracts and assistance agreements (grants, cooperative agreements, etc.). As mentioned in section 4.1, COs are the only individuals who can obligate funds. When procuring services, one should follow the same basic procedure used for the procurement of items. There are certain activities that are of a policy- and decision-making nature that should remain the sole authority of EPA. The CMD should be contacted during the initial planning of the PR to discuss specific requirements for the procurement. The Project Officer (PO) states the service that will be delivered, measures the quality of the service, and accepts the service. When a level-of-effort contract is the vehicle used in procuring services, the work assignment manager (WAM) provides the technical expertise for the work assignment and assumes responsibility for the QA requirements assigned to the PO. Completion of contract administration training is required prior to an individual's being designated a WAM. The Contracting Office is the means of obtaining a contract and of enforcing the provisions. The PO has overall responsibility to see that the service is provided but works through the CO's authority. The PO is appointed by the CO and is formally designated a technical representative of the CO in the contract. POs must complete PO and contract administration training to serve on a contract. Chapter 7 of the EPA Contracts Manual specifies the required training, experience, and workload limitations for an individual to serve as a PO. OAQPS will adhere to these specifications. Two major tools to ensure that adequate service is provided are a well-defined statement of work (SOW) and a QA Project Plan (QAPP) that includes reviews (audits). The QAM or DQAO assists in this activity by providing knowledge and guidance on the QA requirements and aspects of any potential project. The QAM or DQAO will also approve the QA review form that is discussed in the next section. 4.2.1 Contracts Contracts are used when the government derives sole benefit from a particular product or service. Contracts can be specific and can require a degree of lead time for development. Depending upon the scope of the service, QA attributes can be developed that must be adhered to under the terms and agreements of the contract. On May 17, 1996, the Director of OAQPS issued the revised QA Form and a guidance document for its use. The QA Form must be used by the PO to determine if the contract will require environmental data operations (EDO) and, if so, the form includes questions that will determine what QA activities will be required. The form must be used when there is an EDO; if there is no data collection, it is required only for levels above the present small purchase threshold. After the completed form is reviewed by the PO, the OAQPS QAM or DQAO must review and sign it. This form can be found in Appendix B of this document. Whenever the government enters into a contract, it is entitled to receive quality service. In order to define and measure this quality, the PO must develop a SOW that will accurately define the minimum acceptable requirements for the service. This is the first step in the procurement process that helps to ensure that services produce results or products of acceptable quality. The PO must succinctly state his or her expectations of the product or service and must be able to communicate this to the supplier. Good communication between the PO and the supplier of a product or service is essential to a mutual understanding of what the expectations are and how quality will be defined. Methods used to determine quality (audits, quarterly interviews, random inspections, etc.) should be explained prior to project implementation so that the supplier will understand how quality will be assessed. Supplement 2 to OMB circular A-76, A Guide For Writing and Administering Performance Statements of Work for Service Contracts, provides good guidance for writing SOWs and on implementing QA ------- OAQPS Section 4 Revision: 01 Date: 5/96 Page 3 of 3 surveillance plans. Another important source of information is the EPA 1900-Contracts Management Manual which specifies all required documents for developing contracts. Part of the procurement process of certain types of large contracts include the use of a technical evaluation panel (TEP). When this form of contracting mechanism is used to solicit contracts in which a significant percent of the cost (> 25%) includes EDO, the TEP must include a QA representative, if possible, a representative from the OAQPS division processing the contract. Part of the TEP responsibilities will include rating each potential contractor against a standard set of criteria. A portion of this criteria can include various assessments such as on- site audits and the analysis of performance evaluation materials. Prior to the solicitation for bid, it must be determined what proportion of the TEP rating will be allocated to QA assessments. It is suggested that a minimum of 5% of the overall TEP rating be allocated to QA. Depending upon the type of contract used to acquire a service, different types of QA methods for determining the quality of product or output may be used. However, in all cases, documentation is essential. POs are responsible for documenting quality on a regular basis. OAQPS personnel must be aware of the "personal services" type of work characterized by an employer- employee relationship between government and contractor employees. These contracts are illegal in EPA. Personal services conflicts arise when government employees assume the right to instruct, supervise, or control a contractor's employee in how he or she performs work. It is the contractor's right to hire and terminate, to assign, and to organize and implement tasks as the contracting organization deems appropriate. OAQPS may tell the contractor what to do within the terms and agreements of the contract, but not how to do it. 4.2.2 Assistance Agreements Assistance agreements are used when both parties (EPA and the group providing the service) derive benefit out of the service. This usually occurs with grants or cooperative agreements where universities or states derive benefits from participating in EDOs. QA requirements are developed for all assistance agreements that include EDOs. OAQPS follows guidelines developed in the EPA Assistance Administration Manual (EPA-5700). Assistance agreement SOWs are usually developed jointly. However, once the SOW is completed, the parties must also agree on the quality standards for assuring the product or service. It is the responsibility of the PO to be knowledgeable of the EPA QA policy and to represent these standards during the development of the projects SOW. Special conditions are usually included in assistance agreements. The PO will list the conditions to which project participants must adhere. One of these conditions relates to QAPPs. Any assistance agreement that includes EDOs must include the following statement: A quality assurance project plan must be submitted within 90 days of this agreement and/or 30 days prior to commencement of any environmental data operations. Implementation dates will be adjusted based upon the above conditions. Costs associated with data collection are not allowable costs until the quality assurance project plan is submitted, nor will costs be reimbursed until the quality assurance program plan is approved. ------- OAQPS Section 5 Revision 01 Date: 6/96 Page 1 of 2 5. Document and Records Management Federal agencies are required to create and preserve Federal records containing adequate and proper documentation of the organization, functions, policies, decisions, procedures and essential transactions of the agency, and all records necessary to protect the legal and financial rights of the Government and of persons directly affected by the agency's activities (44 U.S.C. 3101). Organizations that perform EDOs and management activities must establish and maintain procedures for the timely preparation, review, approval, issuance, use, control, revision and maintenance of documents and records. A document is any volume that contains information which describes, defines, specifies, reports, certifies, or provides data or results pertaining to environmental programs. As defined in the Federal Records Act of 1950 and the Paperwork Reduction Act of 1995 (now 44 U.S.C. 3301), records are: "...books, papers, maps, photographs, machine readable materials, or other documentary materials, regardless of physical form or characteristics, made or received by an agency of the United States Government under Federal Law or in connection with the transaction of public business and preserved or appropriate for preservation by that agency or its legitimate successor as evidence of the organization, functions, policies, decisions, procedures, operations, or other activities of the Government or because of the informational value of data in them.... " The terms documents and records are often used interchangeably. This section will define OAQPS document and records management procedures. 5.1 Elements of th e System OAQPS is developing a records management system to ensure that completed work meets EPA documentation requirements. A draft document entitled OAQPS Records Management Procedures will also be completed that addresses procedures for document creation, maintenance, and disposition. It also establishes the administrative structure of records management. The document will be implemented in FY97 or 98.. OAQPS has also developed a record keeping document entitled OAQPS File Plan. This plan allows OAQPS staff to identify and file new information (study plans, reports, correspondence, etc.) The OAQPS staff are responsible for preparing many different types of documentation, which are included under the following primary categories in the OAQPS Files Plan: Administrative and Management Audiovisuals Financial Management Technical Support Legal & Legislative Technical Reference Personnel Public Affairs Research and Development Privacy Act Permits Vehicles Inspector General Sampling Analysis Budget Program and Project Management Grants, Contracts, and Agreements Correspondence Publications Non-Point Source Regulatory Development Info/Computer Management Payroll FOIA Property Committees Oversight and Audits International Affairs Under these primary categories are subcategories that cover all documents and records prepared or received by OAQPS. Table 5-1 identifies the various OAQPS QA documents. ------- OAQPS Section 5 Revision 01 Date: 6/96 Page 2 of 2 Table 5-1 QA Documents for Document/Records Storage OAQPS has identified an OAQPS records management coordinator responsible for the implementation and maintenance of the OAQPS records management system. This individual is responsible for the following activities: Ensuring training of OAQPS personnel on the use of the filing system Working with OAQPS staff to determine project deliverables that will eventually be filed Ensuring maintenance of current document and records files Developing, implementing, and maintaining a document inventory Ensuring a yearly deliverables review with project leads Document Title Quality Management Plan QA Report and Work plan QA Project Plans QA Report or Data Quality Assessments (DQA) Standards Operating Procedures QA Track QA Form for Contracts Guidance Documents QA Steering Committee Reports Audits Through development of the yearly action plan, document requirements are identified, and leads responsible for the development of the documents are delegated. The project lead will be responsible for: Determining all deliverables under a project (planning documents, progress reports, final reports, etc.) and deliverables that will require document control. This information should be forwarded to the OAQPS document control manager Determining time lines for various stages of the document (outline, draft, final) Determining the appropriate review cycle (internal vs. external review) Determining the appropriate reviewers Registering and ensuring that all documents and records are incorporated into the OAQPS filing system Although it is ultimately the responsibility of the project lead to file all pertinent documents and records, a formal process that allows for an inventory of expected documentation is very advantageous. A list of expected deliverables will be developed by the project lead. This list will serve as the basis for generating unique file numbers for the documents and will serve as the basis for development of a checklist to ensure that all deliverables are available. 5.2 Document Preparation, Review, and Approval. Document preparation, review, and approval will be dependent upon the type of document being produced. For example, an internal document will have different preparation, review, and approval requirements from an external document. The process befitting each document will be determined by the task lead and his/her immediate supervisor. The approval process for some QA documentation is discussed in Section 2. The peer review process is discussed in detail in Section 9. ------- OAQPS Section 6 Revision: 01 Date: 5/96 Page 1 of 2 6. Computer Hardware and Software The Environmental Protection Agency's ability to fulfill its mission is dependent upon a strong information technology infrastructure. Mission objectives rely on an infrastructure that is capable of supporting environmental information and dynamic communication among EPA offices. One of the most critical components of the EPA infrastructure is information technology (IT). The hardware, software, and communications components that are encompassed by IT form the foundation for environmental information and EPA-wide communication. The management of IT, therefore, is critical to the success of the EPA. The Office of Information Resources Management (OIRM) and the National Data Processing Division (NDPD) are responsible for managing the EPA's IT infrastructure and components. In that role, OIRM and NDPD have established IT standards to manage and ensure that IT components integrate properly into the infrastructure. The Aerometric Information Retrieval System (AIRS) is the primary database for OAQPS. The AIRS User's Guide provides detailed information on the system, from points of contact, to codes, to validation of database fields. 6.1 OAQPS Information Management System All information management system development, improvements, and updates will comply with OARM's System Design and Development Guidance, EPA Directive 2182, dated April 30, 1993, to include a systematic and comprehensive dialogue among the data providers, data and system users, and system developers, prior to the design of the system. It is OAQPS policy to work closely with the OIRM on all phases of system development, improvements, and updates. During the operational phases of information management systems, OAQPS will comply with requirements within OARM's Operations and Maintenance Manual, EPA Directive 2181, April 30, 1993. Compliance with the applicable information resource management standards will ensure that all hardware and software configurations are tested prior to use, to guarantee they perform as expected and meet user requirements. 6.2 Hardware and Software Requirements EPA Quality Assurance Requirements for Computer Hardware and Software Systems for Environmental Programs EPA QA/R-10, will establish requirements for quality in the use of computer hardware and software. This is a planned item and there is no draft currently available. The document will be developed jointly by ORD (QAD) and the OIRM. Target availability is undetermined. EPA QA/G-10 Guidance for Implementing Quality Assurance Requirements for Computer Hardware and Software Systems for Environmental Programs will provide non-mandatory guidance for assuring quality in the use of computer hardware and software. This is a planned item and there is no draft currently available. The document will be developed jointly by QAD and the OIRM. Target availability is undetermined. In addition to OARM's System Design and Development Guidance and Operations and Maintenance Manual, OAQPS will comply with OARM's Delegation of Procurement Authority Guide. This will insure that purchased software will meet user requirements and will comply with IRM standards. ------- OAQPS Section 6 Revision: 01 Date: 5/96 Page 2 of 2 OAQPS intends to comply with the above guidance and expects that appropriate requirements and guidance for hardware, software, and configuration testing will be included in the upcoming EPA QA/G-10 and EPA QA/R-10. 6.3 Data Standards All Federal agencies are required to adhere to Federally mandated data standards and regulations. It is the policy of OAQPS to comply with all applicable regulations, guidance, executive orders, and internal policy documents concerning data standards. These include: The EPA's information data standards and regulations appear in the Catalog of Data Policies and Standards, U.S. EPA, Office of Information and Resources Management, July 1991. It is the responsibility of each individual Program Office to be aware of the current standards and regulations. The National Institute of Standards and Technology (NIST) develops standards and guidelines to achieve the most effective use of Federal information. The Federal Information Processing Standards (FIPS) are the Federal data standards for all data exchange among agencies. Applicable FIPS are listed in the draft EPA document Agency Catalog of Data Policies and Standards, 2IM-1019, July 1991. The EPA Data Standards Program is established and documented in the Information Resources Management Policy Manual. Within EPA, adherence to data standards policy is accomplished through the direction of the OIRM. In general, EPA's data-related policies apply to all EPA organizations and personnel, including contractors, grantees, Senior Environmental Employee Program participants, fellows, and other personnel assigned to EPA who design, implement, and maintain information management systems for OAQPS and EPA. ------- OAQPS Section: 7 Revision: 03 Date: 6//96 Page: 1 of 3 7. Quality Planning It is OAQPS policy to plan its programs and projects effectively. Quality planning must occur at three levels to ensure that OAQPS meets its programmatic and quality goals: (1) Office-wide, (2) program-specific, and (3) project-level. OAQPS participates in Agency-wide planning through its contribution to the OAR operating plan. 7.1 Office-wide Plan n ing The OAQPS operating plan, developed by OAQPS and its divisions planning staff, is the foundation upon which all programmatic activities and corresponding EDOs are based. Annual program plans, tied to the budget process, identify the types of EDO that should occur. OAQPS provides input into the Office of Air and Radiation (OAR) information resources management (IRM) strategic plan. The IRM plan reflects the Office-wide goals outlined in the OAQPS operating plan. The plan identifies the automated data processing hardware, software, full time employees, and type and quality of data needed to meet the OAQPS QA objective. The plan also identifies annual program priorities. As part of the QAARWP, OAQPS will outline planned operational activities in those areas where the organization will focus its quality management efforts for the upcoming year. OAQPS determines those areas on which it will focus its efforts by reviewing activities from the previous year. Based on this review and on the available budget, OAQPS will include in the QAARWP plans to correct any deficiencies in its quality assurance activities. OAQPS must increasingly coordinate the collection and use of environmental data and related activities across many EPA, Federal, state, local, academic, and private organizations. This close coordination is essential to ensure that data are of known type and quality and can be shared where DQOs are similar. 7.2 Program-specific Planning Programs are functional work areas authorized by statute and Congressional direction. The OAQPS programs covered by this QMP include the following: attainment and maintenance of National Ambient Air Quality Standards regulation of Hazardous Air Pollutants permit programs and New Source Review information transfer technical support 7.3. Project-level Planning A project is an organized set of related activities within a program. When a program begins a project, the initiating program will organize a project team. If the project involves environmental data operations, the team should include members who have knowledge or experience in the following areas: sampling, analysis, statistics, and QA/QC. It is the responsibility of the team leader and his/her supervisor to ensure that these areas of expertise are adequately represented on the team. All projects will be subject to the same requirements, whether they are done extramurally or intramurally. ------- OAQPS Section: 7 Revision: 03 Date: 6//96 Page: 2 of 3 As discussed in Section 2 above, the DQO process should be the primary tool used by the team to plan projects. The DQO process will be used to answer the following questions. 1. What is the problem, and how does it relate to the OAQPS mission? Oral statements of the general problem are narrowed into succinct questions that are unambiguous and can be answered with specific data. 2. Once the questions are defined, what are the variables to answering the questions? Use the smallest sets of variables necessary to answer the specific questions raised in question 1. Assemble the variables into precise objectives that illustrate how the measurements will be made to answer the question. 3. What is the allowable level of uncertainty that still enables the question to be answered? This step is necessary for the development of sampling design (e.g., where to sample, how many samples to collect, methods of analysis, etc.) This step is also necessary to develop a QA program to reduce the uncertainty to allowable limits. 4. Who are the customers, and what are their expectations? The plan must identify what types of information are needed (e.g., summary information, detailed trends, graphs, etc.) This information will help the project leaders in focusing the project objectives and in determining the required data quality. 5. Who are the suppliers, and what are their responsibilities? Details on the organizations participating in the project and on their responsibilities are required to ensure that important phases and operations of the program are not overlooked. Project phases normally include the following: management, design, planning and budgeting, implementation, method development, information management, reporting, and QA. Planning documentation should identify the personnel responsible for these phases, and these individuals should be included in all planning meetings. Documentation of the project is critical and is discussed in detail in Section 5 above. While the data from a project may be technically sound, inadequate documentation can cast doubt on the data and make it difficult, if not impossible, to defend. The documentation must address all phases of the EDO. All EDOs should include a work plan, a QAPP, a QA report, and a final report. The work plan would include information on the project phases mentioned above; the QAPP would include information presented in Section 2; the QA report would provide an assessment of the data quality (as documented in the QAPP); and the final report would provide ------- OAQPS Section: 7 Revision: 03 Date: 6//96 Page: 3 of 3 information satisfying the project objectives. The QA documents do not necessarily have to be separate documents but can be included in the study plan (QAPP) and the final report (QA report). As part of project planning, project leads will develop time lines for the development, review, and completion of required documentation. OAQPS is committed to the Agency policy of peer review. The OAQPS peer review procedure is described in detail in Section 9.6 below. Team Leaders should be familiar with this policy. The Team Leader should also identify appropriate reviewers of the documentation. When project teams acquire secondary data, the data must be qualified before they are used. Secondary data are data from sources other than EPA or are data collected by EPA that are being used for a purpose other than that for which they were collected. The DQO process will be used to qualify secondary data. The project team must answer the following questions: What kinds of decisions will be based on these data? Data may form the basis for an emission standard or an emission factor or may be background information. What are the key data that drive the decision? The secondary data may be the only available data or may merely supplement existing data collected by EPA. What is the allowable level of uncertainty? This step is necessary to develop the design for the data assessment. After establishing the acceptable level of uncertainty, the appropriate statistical analyses will be done to determine if the data are acceptable. ------- OAQPS Section 8 Revision: 01 Date: 6/96 Page 1 of 2 8. Implementation of Work Processes The procedures described in this Section on the implementation of work process must be followed within all Divisions of OAQPS. Within this Section, the implementation of programs will be discussed through the use of the Quality Management Plan (QMP), the Quality Assurance Annual Report and Work Plan (QAARWP), and Standard Operating Procedures (SOPs), with the proper levels of management participation and approval identified. Project implementation will also be considered by focusing on the implementation of the Quality Assurance Project Plan (QAPP). The QMP will be a living document that is revised annually to reflect any changes in policy and/or implementation of OAQPS's procedures. Full review of the QMP, with approval from the Quality Assurance Division (QAD), will occur every five years, with yearly changes only receiving OAQPS management approval and QAD notification. Any revisions that occur to the QMP will be documented, and notice will be given to all OAQPS staff. More detailed information about specific areas of concern are addressed in the other sections of this document. 8.1 Program Implementation OAQPS developed the QMP as a means of documenting how an organization will plan, implement, and assess the effectiveness of quality assurance (QA) and quality control (QC) operations applied to environmental programs. All Divisions within OAQPS are responsible for their implementation of the QMP. The level of management required is decided by program managers within a program area. Areas that need further development or consistency shall be addressed through SOPs. A SOP is a written document that details the method for an operation, analysis, or action with thoroughly prescribed techniques and steps, and that is officially approved as the method for performing certain routine or repetitive tasks. Further details on SOPs can be found in section 2.1.5 of this document. SOPs will be developed by subject-specific workgroups with required levels of staff and management approval. They should use the Agency guidance EPA QA/G-6) Guidance on Standard Operating Procedures for Environmental Data Operations to ensure consistent conformance with organizational practices, serve as training aids, provide ready reference and documentation of proper procedures, reduce work effort, reduce error occurrences in data, and improve data comparability, credibility, and defensibility. All EPA organizations conducting environmental programs that have a QMP must submit an approved QAARWP to the director of the Quality Assurance Division (QAD) by November 1 each calendar year. The purpose of the QAARWP is to inform Agency senior management about the status and effectiveness of the organization's QA program. The QAARWP documents the findings of management's assessment of the organizations's Quality System, documents performance during the immediate past fiscal year, and provides the work plan for the upcoming fiscal year's priorities for the organization's quality system. The quality system is a structured and documented management system describing the policies, objectives, principles, organizational authority, responsibilities, accountability, and implementation plan of an organization for ensuring quality in its work processes, products (items), and services. 8.2 Project Implementation It is OAQPS policy that all decisions and work involving the use of environmental data be supported by a QAPP. The QAPP is a formal document describing in comprehensive detail the necessary QA, QC, and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance criteria. The QAPP provides a project/task-specific blueprint of how QA and QC are applied ------- OAQPS Section 8 Revision: 01 Date: 6/96 Page 2 of 2 to an environmental data operation to assure that the results obtained are of the type and quality needed and expected. OAQPS will use the QAPP for the implementation of work associated with environmental data collection. The Work Assignment Managers (WAMs) and DQAOs, in this order, are responsible for approval and implementation. Their immediate supervisors are responsible for ensuring the WAMs and QAOs perform these duties in compliance with the specific regulations and guidance. ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 1 of 8 9. Assessment and Response The Office of Air Quality Planning and Standards uses assessments to evaluate and improve the quality of environmental data operations. The assessments are an independent process of evaluating the ability of an organization to function as documented. The assessments help ensure the integrity of environmental data collection programs. These collected environmental data are the basis for regulatory and guidance development and for compliance assessment, across and the entire Agency. OAQPS plans to use EPA QA/G-7 Guidance for Determining Quality Training Requirements for Environmental Data Operations to develop training requirements for those involved in assessment and response activities, including management systems reviews and audits. Until that guidance is available, the OAQPS Supervisors must use their judgement to ensure that each time they assign someone to an assessment and response activity, the individual assigned has the correct technical expertise; and the appropriate audit and/or management systems review training. 9.1 Management Systems Reviews A management systems review (MSR) is the qualitative assessment of a data collection operation and/or an organization(s) to establish whether the prevailing quality management structure, policies, practices, and procedures are adequate for ensuring that the type and quality of data needed are obtained. It is also an on-site evaluation by the organization's senior management to assess the organization's internal management structure and its documents to determine whether the organization is implementing a satisfactory QA program. It is used to determine the effectiveness of, and adherence to, the QA program and the adequacy of resources and personnel provided to achieve and ensure quality in all activities. The MSR includes reviews of: Procedures for developing DQOs. Procedures for developing and approving QA project plans (QAPPs). The quality of existing QAPP guidance and QAPPs. Procedures for developing and approving standard operating procedures (SOPs). Procedures and criteria for designing and conducting audits. Tracking systems for assuring that the QA program is operating, and that corrective actions disclosed by audits have been taken. The degree of management support. Responsibilities and authorities of the various line managers and the quality assurance program manager for carrying out the QA program. Those OAQPS personnel involved in MSRs are recommended to take the management systems review course offered through QAD. 9.1.1 OAQPS MSR An MSR of the QA program will be conducted at a minimum of once every three years; and more frequently if serious deficiencies exist. The review should occur between the months of March and May, to allow results to be documented in the Quality Assurance Annual Report and Workplan (QAARWP). Therefore, planning for the review should begin in January or early February of the year of the review. In order to achieve the MSR's objectives, the review should be conducted by an individual somewhat independent of the organization but still with a "stake" in seeing improvement. Since OAQPS cooperates with ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 2 of 8 many agencies, a lead could be selected from one of the cooperators. The lead could then choose a review team from OAQPS senior management who would assist in planning, scheduling, and implementing the review. The review team would determine the scope of the review, which would include reviews of the bullets mentioned above. An audit plan should be developed. The team will present its findings in a report directed to OAQPS management. Information on the results of the MSR will also be reported in the QAARWP. Action items on any deficiencies will be developed and discussed in this report and will become goals for improvement. Review of progress on actions items will be discussed at management meetings and subsequent QAARWPs. 9.1.2 External MSRs Various groups within OAQPS work cooperatively with the EPA Regions in order to implement major data collection activities (i.e., Ambient Air Monitoring Program). Group leads will be responsible for determining what MSRs are required, and at what frequency they should be implemented. Each year, the QAM will be responsible for contacting group leads to determine the MSRs that will be conducted and to incorporate this information into the QAARWP. External MSRs will be conducted in the same manner as the internal OAQPS MSR. 9.2 Surveillance Surveillance is a continual, or at least frequent, monitoring and verification of the status of an entity and an analysis of records to ensure that specified requirements are being fulfilled. It occurs when the product user oversees the actions of the producer, on a real-time basis, during development of the product. One purpose of surveillance is to identify potential problems as quickly as possible, and to institute corrective action such that a suitable product is developed for the user. Effective use of surveillance will reduce the negative impact on both producer and user in developing an acceptable product. A traditional environmental data collection scenario is assumed for illustrative purposes; however a similar producer/user concept should be used for other OAQPS products. There are two types of problems which can be uncovered during surveillance. The first type of problem is failure of the producer to adhere to the previously agreed-upon quality measures documented from the DQO process. This type problem should be infrequent, but can be costly. It is the responsibility of the surveillance personnel to promptly notify the producer's on-site representative of the specific quality measure allegedly violated, and to explain why continued violation could adversely affect product quality. It is the producer's responsibility to confirm the violation and to propose corrective action. If there is failure to agree that a violation has occurred, it is still the producer's responsibility to propose and explain why the alternative will be acceptable, and to document corrective action acceptable to the user. For a traditional data collection effort, one corrective action is to abort and come back another time. Other, less severe, alternatives are usually more cost effective. The second type of problem identified through surveillance is the discovery of additional technical refinements that would make the product more useful or less costly to produce. It is the responsibility of both the producer and the user representative to discuss these type of problems as time permits. It is the responsibility of the person conducting the surveillance to document these discussions and to submit suggestions to both user and producer management for improvement of future similar projects. In general, unless technically required to yield an acceptable product or planned for in advance, the data gathering effort should not change "on the fly". The negative effect caused by upsetting effective staff training, equipment availability, and staff scheduling in gathering the data usually outweigh, the benefits to improved product quality, if the data collection effort has been adequately designed. ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 3 of 8 It is the responsibility of the surveillance user to determine when surveillance is justified. Surveillance is a labor-intensive and costly tool. Accordingly, the surveillance tool should be reserved for sensitive data- gathering projects. Data gathering can be sensitive for technical reasons, such as those involving large numbers of critical simultaneous measurements, establishing correlations with key process operations, or using experimental test methods where future refinements are likely to be needed. The data-gathering effort may also be sensitive for political reasons, such as an uncooperative host site or no clear consensus among stakeholders as to what the data will represent. It is the duty of surveillance user staff to estimate the cost and to identify the technical and perceived political situation. It is the responsibility of the user group leader, acting in conjunction with OAQPS senior management, to confirm the political implications of the project, to provide a mechanism to balance the potential benefits with potential benefits of spending the resources on other projects, and to commit appropriate resources. Because the effective use of surveillance is largely based on the subjective evaluation of the person conducting the surveillance, it is important that the person chosen have sufficient knowledge of both the production process and the product application process. In OAQPS environmental data operations that typically support regulatory development, this means a knowledge of test methods being applied, ability to analyze the significance of unanticipated deviations from the agreed data collection plan due to deviations from test methods or process operation, skill in matching a fixed budget with desire to obtain complete data sets, and skill in effective verbal communication. It is a user and producer management responsibility to establish a systematic method of choosing qualified surveillance personnel. 9.3 Technical Systems Audits Technical systems audits (TSAs) are a thorough, systematic, on-site, qualitative audit of facilities, equipment, personnel, training, procedures, record keeping, data validation, data management, and reporting aspects of a system. TSAs are also qualitative on-site evaluations of a complete phase of an EDO (i.e., sampling, preparation, analysis). This audit can be performed prior to the data collection activity, in order to verify the existence, and to evaluate the adequacy, of equipment, facilities, supplies, personnel, and procedures that have been documented in the QAPP. TSAs are also employed during the data collection activity in order to verify and evaluate the EDO. OAQPS will perform TSAs for all EDOs of category 1 and 2, as defined in Section 2 of this document. The number and frequency will be dependent on the length of the project and on the evaluations of prior audits. The frequency should also be described in the contract scope of work and/or work assignment and should be described in the contract QA Form, in order for the organization performing the work to have knowledge that this type of activity will be performed. A TSA at the beginning of each project is advantageous. The audit should be conducted approximately one week after start-up, to allow the data collection personnel to develop a routine. Audits will be scheduled by the Program Manager/WAM who is responsible both for developing an audit plan and for documenting audit results. 9.4 Audit Plan Audit planning is a necessity in order to conduct efficient audits. An audit plan for all types of audits will include the following items: Audit title Audit number- Year and number of audit can be combined, as in 91-1,91-2 ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 4 of 8 Date of audit Scope - Purpose - Standards ¦ Audit team - Auditees - Documents - Timeline ¦ Establishes the boundary of the audit, and identifies the groups and activities to be evaluated. The scope can vary from a general overview, to total system, to part of a system, and will effect the length of the audit What the audit should achieve Standards are criteria against which performance is evaluated. These standards must be clear and concise and should be used consistently when auditing similar facilities or procedures. The use of audit checklists are suggested to assure that the full scope of an audit is covered and provides consistency when auditing the same activity more than once Team lead and members People who should be available for the audit from the audited organization. This should include the program manager, principal investigator, organization QA representative(s), other management, and technicians as necessary. Documents that should be available in order for the audit to proceed efficiently. Too often,documents are asked for during an audit, when auditors do not have the time to wait forthese documents to be found. Documents could include QMPs, QAPPs, SOPs,GLPs, control charts, raw data, QA/QC data, previous audit reports, etc A timeline of when organizations (auditors/auditees) will be notified of the audit in order for efficient scheduling and full participation of all parties. The audit plan document is not a major undertaking and, in most cases, will be a one-page table or report. However, the document represents thoughtful and conscious planning for an efficient and successful audit. The audit plan should be made available to the organization audited, with adequate lead time to ensure that appropriate personnel and documents are available for the audit. 9.4.1 Audit Reporting A debriefing will occur at the completion of the audit. Positive and negative aspects of the audited activity will be discussed among the audit team and management of the area audited, and, if necessary, the technical personnel performing the measurement activity. Copies of the draft audit summary and findings should be provided to all those in attendance. Necessary action to improve the measurement system/organization will be discussed with audit participants. In the case of TSA, DQAs and PEs, the reporting responsibility rests with the Program Manager/WAM, though he/she may not be the review team lead for the audit. Reporting MSRs is the responsibility of either the review team lead or an appointed designee. The report will include: Audit title and number and any other identifying information Names of audit team leaders, audit team participants and audited participants Background information about the project, purpose of the audit, dates of the audit, particular measurement phase or parameters that were audited, and a brief description of the audit process Summary and conclusions of the audit and corrective action Attachments or appendices that include all audit evaluation forms and audit finding forms The audit report should be completed within five working days of completion of the audit. TSA, DQA, and PE reports will be reviewed, signed by the Program Manager/WAM and Project Officer, and filed according to the information in Section 5 above. MSR reports will be reviewed, signed by the audit lead and the OAQPS Director and filed both in the Director's office (see Section 5) and with the QAM. It is the responsibility of the review team lead to forward audit reports to the appropriate participants. The audit report should have restricted ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 5 of 8 distribution, in order to foster constructive working relationships. When significant concerns are identified on audit-finding forms, a meeting will be scheduled with the appropriate parties. 9.4.2 Response Actions The audit reports will be discussed with the audited organization, and action(s) necessary to rectify and control the situation will be developed. Line management may be asked to assist in problem resolution, as necessary. For each audit-finding form, an audit-finding response form will be developed to track corrective actions. This information will be included in the audit file. The Program Manager/WAM Officer (TSAs, DQAs PEs) or the OAQPS Director (MSRs) is responsible to ensure compliance with the corrective actions. If major deficiencies are found, follow-up audits may be required and should be discussed. 9.5 Performance Evaluations A PE is a type of audit in which the quantitative data generated in a measurement system are obtained independently and are compared with routinely obtained data, to evaluate the proficiency of an analyst or laboratory. It is also a way of testing (evaluating) a laboratory's or field collection ability to accurately determine the concentration of environmental monitoring samples. PE programs are developed as a tool to help ensure the quality of the Agency's environmental data operations. PE programs are important because environmental data are used as a basis for regulatory and guidance development and for compliance assessment across the Agency. PE programs are strongly supported and used by the Regions, States, and local agencies. PE studies will be a key component of the National Environmental Laboratory Accreditation Program now being developed by the Agency-wide Environmental Monitoring Management Council. The PE program consists of ambient air and source emission evaluations. OAQPS will send a participant a sample of unknown concentration. The participants perform a routine analysis during standard operations and report back to OAQPS this assessment of the sample. The participants value is compared with the true certified (ORD, NIST, etc.) value. If the difference is outside of a stated known quality assurance acceptable limit, the participant is asked to further evaluate this performance for any known deviations or abnormalities. If no reasons are found for the deviations, the data are identified as being outside of the stated known quality assurance range for that particular pollutant. 9.5.1 Ambient The National Performance Audit Program (NPAP) is a national program required by regulation and consisting of approximately 4,835 ambient air pollution monitors in the ambient air network comprising the State and Local Air Monitoring Stations (SLAMS), National Air Monitoring Stations (NAMS), Photochemical Assessment Monitoring Stations (PAMS), and Prevention of Significant Deterioration (PSD) sites. The PAMS were added to NPAP in 1995. The NPAP audits are accomplished using a variety of mailable audit systems. The participants use these audit systems to generate pollutant concentrations and flowing air streams, which are then introduced into their sampling system. The pollutant concentrations and air stream flow rates are unknown to the audit participants. The outputs from the sampler that result from the use of the audit system are recorded on a data form, returned to EPA, and compared with the concentration or flow rate that should have been generated by the audit system under the environmental conditions at the site. The differences between the EPA expected (certified) values and the NPAP participants reported values are calculated and are returned to the participant. The NPAP is operated by the National Exposure Research Laboratory (NERL) with technical contract support. The NPAP's goal is to provide audit materials and devices that will enable EPA to assess the proficiency of agencies that are operating monitors in the SLAMS, NAMS, PSD, and PAMS networks. To accomplish this, NPAP has established acceptable limits or criteria for each of the audit materials and devices ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 6 of 8 provided in the program. All audit devices and materials used in the NPAP are certified as to their true value, and that certification is traceable to a NIST standard material or device, wherever possible. 9.5.2 Source Emission On November 15, 1990, the Clean Air Act (CAA) was amended, and a list of 189 compounds that are considered to be Hazardous Air Pollutants (HAPs) was included. These pollutants are to be regulated through the development of Toxics Methods, which include MACT (Maximum Achievable Control Technology) Standards and New Source Performance Standards (NSPS). In support of these Federal regulations, audit materials are developed, validated, and provided to state and local agencies to ensure high quality source emissions compliance data. These performance evaluation samples have traditionally been requested from the Stationary Source Compliance Test Coordinator of EPA's National Exposure Research Lab in Research Triangle Park , NC by the regulatory agency for whom the compliance test is being conducted. When the request is received, the coordinator selects the audit materials to be used and directs the contractor maintaining the inventory of audit materials to send the audit samples to either the requesting organization or to the test site. The samples are sent via ground transportation with all the shipping costs prepaid by the EPA contractor. The organization being audited takes a gas sample from each audit cylinder using its Method sampling system and analyzes them, as unknowns, in the same manner as its stack samples. At the completion of the audit, the audit results are returned to the EPA WAM by the regulatory agency representative, and the audit cylinders are returned to the contractor responsible for maintaining the inventory. The development of source emissions methods is currently being converted to an in-house research program supported by ORD scientists. The current "acceptable" validation of a source emissions method includes the validation of the method first in the laboratory and then at two field locations with different source emissions. This process, depending of species of interest, is resource intensive for ORD support and will require an agency initiative to provide necessary resources. 9.6 Audits of Data Quality Many important EPA decisions are based on the nationwide monitoring data obtained by the state and local agencies. Data collected and reported to the Agency's Aerometric Information Retrieval System (AIRS) are used by the EPA to aid in planning the Nation's air pollution control strategy and to measure achievement toward meeting National Ambient Air Quality Standards (NAAQS). Further, the data in AIRS are made available to numerous requestors, who may use the data for various research projects, special studies, or other purposes. The quality of national monitoring data should be determined and made known to all data users to assure their most knowledgeable and effective use. The Code of Federal Regulations (CFR), Part 58, directs that precision and accuracy checks be incorporated by state and local agencies to verily the quality of the collected data. Precision is defined in 40 CFR Part 58, Appendices A and B, in the sense of "repeatability of measurement values under specified conditions". Accuracy is defined in 40 CFR Part 58, Appendices A and B, in the sense of a measure of "closeness to the truth". Section 3 of Appendix A requires that measures of data quality be reported on the basis of "reporting organization"; these are defined as either a State or a subordinate organization within a State that is responsible for a set of stations monitoring the same pollutant and for which precision and accuracy assessments can be made. States must define one or more reporting organizations for each pollutant, such that each monitoring station in the State SLAMS network is included in one, and only one, reporting organization. ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 7 of 8 The stated guideline for determining compliance with precision and accuracy requirements is found in the Quality Assurance Handbook, Volume 2, Section 2.0.11, which states, "As a goal, the 95% probability limits for precision (all pollutants) and TSP accuracy should be less than + 15%. At 95% probability limits, the accuracy for all other pollutants should be less than + 20 %." The precision and accuracy data are found in the EPA AIRS Air Quality Subsystem, precision/accuracy reporting organization summary report for criteria pollutants. The precision and accuracy data are submitted to the EPA AIRS and are categorized by Reporting Organizations (RO,) and then further categorized by Region and State. The precision and accuracy data can be disaggregated by year and then by subsequent quarters of years. OAQPS is developing a program for incorporating the use of the these data in an overall assessment of data quality. 9.7 Peer Review Peer review is well established as a mechanism for assuring the quality, credibility, and acceptability of both individual and institutional work products of technical and scientific nature. OAQPS utilizes two integrated mechanisms for implementing effective peer review, its Standard Operating Procedures (SOPs) document and a Peer Review Implementation Work Group. The SOP document formalizes the procedures that OAQPS uses for peer review of scientific and technical work products. This SOP is intended to guide the peer review process in a way that maximizes the very real benefits of ensuring that OAQPS products are founded on sound science and engineering principles while not establishing impediments for timely development and issuance of work products. Appendix C of the SOP lists (by peer review category) representative technical activities/products produced by OAQPS over the past three years. Appendix D records the specific peer review decisions made in OAQPS over the past year; this list is updated annually. An attachment to the SOP, entitled Alternative Peer Review Approaches for Survey, catalogs a variety of approaches for obtaining both peer review and peer involvement for scientific and technical work products associated with regulatory and other policy decisions at EPA. Supplementary information on peer review procedures for major scientific or technical products defined in the SOP can be found in the Office of Air and Radiation Standard Operating Procedure for Peer Review of Major Scientific and Technical Documents. The OAQPS Peer Review Implementation Work Group was created to oversee implementation of the OAQPS SOP within each division; periodically to review and, as needed, to make recommendations for modifying the SOP; and to serve as principal liaison between the Science Policy Council and OAQPS in relation to peer review issues. Members of the OAQPS Peer Review Implementation Work Group are identified in Appendix B of the OAQPS Peer Review SOP. Specific responsibilities of the members are denoted in Section V.B of the SOP. 9.8 Readiness Review Readiness is a predictive measure of ones ability to deliver a product or service in a timely and effective manner at some future time. Unlike surveillance, which focuses on whether the agreed-upon events are actually happening, readiness review focuses on the preliminary events that, if complete, are assumed to be strong indicators that the main event (e.g., a data collection activity) has a reasonable chance of being delivered successfully. Hence in OAQPS, readiness reviews are most likely effective when sufficient similar products are delivered to allow development of a checklist of key prerequisites. Because the effective use of a readiness review is largely based on the subjective evaluation of the person conducting the review, it is important that the person chosen have sufficient knowledge of the production process and of the product application process. It is the producer management responsibility to establish a systematic method of choosing qualified personnel. It is the responsibility of the personnel conducting the readiness review to report findings to the producer management. It is the responsibility of the producer ------- OAQPS Section 9 Revision 01 Date: 6/96 Page 8 of 8 management to assure that corrective action is developed and implemented when indicated. 9.9 Data Quality Assessment Data quality assessment (DQA) is the statistical analysis of environmental data, to determine whether the quality of data is adequate to support the decision. Data are appropriate if the level of uncertainty in a decision based on the data is acceptable. The DQA process is described in detail in Guidance for the Data Quality Assessment Process, EPA QA/G-9 (Final Draft due Summer 1996) and is summarized below. 1. Review the data quality objectives (DQOs) and sampling design of experiment (DOE): review the DQO and develop one, if it has not already been done. Define statistical hypothesis, tolerance limits, and/or confidence intervals 2. Conduct preliminary data review. Review Precision &Accuracy (P&A) and other available QA reports, calculate summary statistics, plots and graphs. Look for patterns, relationships, or anomalies 3. Select the statistical test, select the best test for analysis based on the preliminary review, and identify underlying assumptions about the data for that test 4. Verify test assumptions', decide whether the underlying assumptions made by the selected test hold true for the data and the consequences. 5. Perform the statistical test: perform test and document inferences. Evaluate the performance for future use This is an iterative process in that, if at any step the decision is not to go to the next step, then the previous steps are revisited. ------- OAQPS Section 10 Revision: 01 Date: 6/96 Page 1 of 2 10. Quality Improvement The intent of this Section of the QMP is to develop and foster the concept that quality assurance is the philosophy that, in all work functions, OAQPS should be continually striving for improvement. This Section describes how OAQPS will detect and prevent quality problems and describes the process for ensuring continuous improvement. This process will be discussed for both program and project levels. 10.1 Program Review and Improvement OAQPS Management has recognized the importance of program review and, with its last reorganization, has incorporated a Program Review Group into the Information Transfer and Program Integration Division. The Program Review Group manages the review of major air programs that cut across Division, Office, and Agency lines. The Group establishes criteria for evaluating programs and works closely with other OAQPS Groups and programs, as well as other Headquarters, Regional, state/local and other organizations involved, to determine the scope of benchmarks and measures of progress. The Group works closely with all levels of the air program, to assess program strengths and weaknesses and to assist in the development and implementation of program improvements. In addition, the QA Team meets regularly to discuss cross-cutting issues and to look at ways of improving the organizational philosophy about QA. Of particular benefit is the representation of all the Divisions within OAQPS on the QA Team. This provides unique views of the QA process and of what programs it applies to. By discussing new aspects of QA, the Team has the opportunity to continuously improve the individual programs. OAQPS Management will rely on the regular internal program reviews undertaken by the Program Review Group and on the issues identified by the OAQPS QA Committee as key mechanisms to identify areas of concern. OAQPS Management will also encourage the QAD to conduct Management System Reviews at least every three years as an external program review. Any deficiencies identified through any of these mechanisms will be addressed through the development of appropriate action plans, which could include revision of the QMP, if necessary. The QMP is reviewed by the QA Team in order to determine if the document remains relevant to the OAQPS mission. The Quality Assurance Annual Report and Work Plan (QAARWP) serves as the assessment mechanism and the blueprint for the next year's actions at improving programs and at implementing any QMP revision needed to address changes to the OAQPS mission . 10.2 Project Reviews and Improvement Project reviews can be accomplished using the tools described in Section 9. There are two types of project reviews. The goal of the first type of project review is to detect and correct conditions that could adversely compromise the ability to use the products for its intended purpose. This goal will be accomplished by employing the principals of data quality objectives in the planning phase of the project, to translate the user's needs into defined product characteristics that, if met, both user and producer agree will satisfy the user's need. This will be documented and will form the basis for determining the success of the project. It is the duty of the project staff lead responsible for producing the product to assure that this documentation is developed and its goals met. For traditional environmental data operations, this is typically called a QAPP, or Site Test Plan. For products such as a regulation, this simile should be emulated, but the measures of success will usually be more global and will include policy measures as well. It is the duty of the user staff lead to review and (ultimately) ------- OAQPS Section 10 Revision: 01 Date: 6/96 Page 2 of 2 approve the proposed specifications prior to the start of the project, to provide timely notification to both the producer and his/her Group Leader if project needs change that could require product changes, and to promptly notify the producer if there is a problem with using the finished product. It is the user Group Leader's responsibility to resolve potential budget increases that result. Where the producer and user are both in OAQPS, it is the responsibility of the producer and user Group Leaders to assure that a systematic process is in place to accomplish the goal of timely detection and correction of conditions that could adversely compromise the product's end use. It is the responsibility of the OAQPS senior management and Division Directors to provide a work atmosphere that encourages this process and, when the user is outside of OAQPS, to take on the role of key user (decision maker) during the planning phase, to assure that the measures by which success of the project will be judged are conveyed to the project lead. There will be a data quality assessment made at the end of the project, to assure that the measures of success are met and to provide input to the second type of project review. The goal of the second type of project review is to look for ways to assure that the customer receives future products within the required time frame, at acceptable quality, and for less cost (continuous process improvement). It is the responsibility of the OAQPS senior management and Division Directors to provide both a work atmosphere that encourages process improvement and a budget system that documents and rewards cost- saving innovation. It is the responsibility of the Group Leader to assure that a systematic process is implemented that categorizes typical work products of the group, documents historical costs, assesses the potential for process improvements, and assures the implementation of improvement recommendations resulting from this systematic process. It is the responsibility of the project staff to actively participate in this systematic process by offering suggestions and by implementing recommended process improvements. ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 1 of 17 Appendix A Group Functions The following provides a brief description of the functions of the various groups in the OAQPS divisions, including the Office Management. This information was developed in the spring of 1994, during the Offices reorganization. ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 2 of 17 OAQPS Main Office Staffs Washington Operations Staff The Washington Operations Staff, under the supervision of the Staff Director, serves as the primary liaison between EPA headquarters and the OAQPS Immediate Office, coordinating information and communication on issues of importance to OAQPS and Agency managers, representing OAQPS in meetings and discussions, and ensuring timely processing of OAQPS products in Washington. Specifically, the Staff: Facilitates communication and information flow on program and policy issues between Washington HQ and OAQPS Tracks and ensures OAQPS policy, regulatory, and other key packages proceed through review process in a timely fashion Serves as OAQPS representative when necessary in meetings and discussions in Washington with officials from EPA Headquarters, other Federal agencies, industry, and State, national and international organizations Monitors and analyzes Agency activities which affect OAQPS programs and policies Planning, Resources, and Regional Management Staff The Planning, Resources, and Regional Management Staff, under the supervision of the Staff Director, manages the development of program goals, objectives and plans and develops systems for monitoring and assessing program progress, including Headquarter, Regional, State & local resource and program management systems. It manages program resources, including budget, contracts, and human resources, and coordinates cross- cutting administrative support services. It also facilitates communication between OAQPS and Regions, States and local agencies on program issues. Specifically, the staff: Manages long-range and strategic planning processes Develops and manages OAQPS portions of Headquarters and Regional program progress tracking systems (e.g., Program Specific Guidance, Memorandums of Agreement (MOA)) Manages budget formulation and execution for OAQPS portions of Headquarters & Regional resources and State grants Facilitates communication and cooperation between OAQPS and Regional offices and States and local agencies on resource, planning, and program issues Coordinates regional program reviews Coordinates Office-wide human resource initiatives (e.g., training, development, human resource mini- council) Coordinates OAQPS contracts and other extramural resource management processes and ensures that adequate fiscal controls are maintained Manages office-wide administrative support services ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 3 of 17 Groups under the Environmental Monitoring and Analysis Division Air Quality Trends Analysis The Air Quality Trends Analysis Group, under the direction of the Group Chief, is responsible for tracking and analyzing ambient air quality trends and gathering, producing and distributing information on progress in air quality. Specifically, the Group: Manages the development environmental trends and other statistical analyses, especially for air quality, including Clean Air Act Indicators, the annual Air Quality and Emission Trends Reports, the Air Quality Atlas, and the Ozone Design Value Study Establishes air quality indicators of progress, analyzes air pollution trends, and distributes information on progress in reaching air quality goals Provides statistical and analytical support to other OAQPS elements and program (e.g., National Ambient Air Quality Standards (NAAQS) review process, State implementation plan (SIP) program, emissions measurement) Coordinates with the Office of Research and Development (ORD) on the full range of statistical and data analysis issues to improve base of relevant scientific information Coordinates the development of trends information with other OAQPS offices Monitoring and Quality Assurance The Monitoring and Quality Assurance Group, under the direction of the Group Chief, is responsible for developing new ambient monitoring methods and promoting the use of the highest quality ambient air data in decision-making. Specifically, the Group: Develops national regulations and guidance on air quality monitoring. Develops guidance and oversees the implementation of the photochemical assessment monitoring station (PAMS) network including conducting workshops, developing guidance documents, and providing analytic support. Provides national oversight and direction to Regional and State/local air monitoring through the MOA process, national ambient monitoring stations/State and local monitoring stations (NAMS/SLAMS) coordinators, Standing Air Monitoring Work Group (SAMWG), and the Ambient Monitoring Technical Information Center (AMTIC). Provides technical direction for the development of new monitoring technologies and networks, including toxics monitoring and Fouier Transform Infrared Spectroscopy (FTIR) Coordinates with ORD on the full range of air quality monitoring and quality control/quality assurance issues to improve base of relevant scientific information, including new technologies and field studies. Air Quality Modeling Group The Air Quality Modeling Group, under the direction of the Group Chief, is responsible for providing leadership and direction on the full range of atmospheric dispersion models and other mathematical simulation techniques used in assessing source impacts and control strategies. The Group serves as the focal point on modeling techniques for other EPA headquarters staff, Regional Offices, and State and local agencies. It coordinates with ORD on the development of new models and techniques, as well as wider issues of atmospheric research. Finally, the Group conducts modeling analyses to support policy/regulatory decisions in OAQPS. The Group specifically: ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 4 of 17 Provides guidance to the Regions and State/local agencies on the selection and use of models in regulatory settings. Evaluates and conducts validation studies on new models and simulation techniques for use by Regions and State/local agencies and makes revisions to improve their use in regulatory settings. Coordinates with ORD on the need for development of new models to meet emerging needs (ie. new applications, new NAAQS, etc.) Conducts modeling studies to support policy/regulatory decisions within OAQPS (ie. NAAQS decisions, risk analyses) Through an interagency agreement with the National Oceanic and Atmospheric Administration (NOAA), obtains on-site meteorological expertise and analytical support on the use of dispersion models and other simulation techniques and the review and validation of new and revised models. Emissions Inventory and Factors Group The Emission Inventory and Factors Group, under the direction of the Group Chief, is responsible for providing technical assistance to Regional, State and local clients on the development of emissions inventories and factors, and for coordinating the use of inventories and factors in regulatory decision-making. The Group also identifies emerging needs and coordinates with ORD on research and development related to emissions inventories and factors. Specifically, the Group: Develops national emissions estimates for a variety of policy and programmatic uses such as trends analysis, regional scale modeling and regulatory impact assessments Develops and distributes technical guidance on emissions inventory and factor development for both criteria and toxic air pollutants, including quality control/quality assurance and estimation procedures Manages and coordinates the national SIP inventory effort in support of Title I of the Clean Air Act Amendments of 1990 Evaluates and develops new and revised emission factors to support inventory development and national regulatory efforts Conducts studies leading to the development of new or revised emission factors, including controlled and uncontrolled studies on criteria and toxic pollutant sources and species profiles Develops and coordinates with ORD on new methodologies and techniques to estimate emissions from a variety of sources Provides leadership on the correct use of emissions inventory and factors methodology and information to Regions and State/local agencies through a clearinghouse and other media Provides leadership in the development and use of emission factors through: preparation and dissemination of technical guidance and information; enhancements to and maintenance of reference guides (AP-42) and technical information dessimination tools (e.g., FaxCHIEF, AIRCHIEF, and the CHIEF electronic bulletin board); and training workshops. Source Characterization Group A The Source Characterization Group A, under the direction of the Group Chief, is responsible for a broad range of source characterization activities including emissions testing, methods development, and reviewing emissions methods submitted by other groups. The Group carries out these activities to support the development of national emission standards and implementation of national regulatory programs. It is also responsible for disseminating this and other information to the regulatory and industrial community. This Group works closely with Source Characterization Group B to allocate responsibility for specific source characterization projects. Specifically, the Group: ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 5 of 17 Plans and conducts field test programs to provide quality data in support of regulatory development consistent with data quality objectives Evaluates and reviews test results and methods developed by other groups (both EPA, State/local agencies as well as the private sector), including the review of methods submitted under Method 301 Provides expert evaluation and reports of source characterization data and related documentation in support of regulatory development programs Promotes consistency in the application and use of all source characterization techniques (especially emission test procedures) by providing expert technical assistance, including reviewing modifications and alternatives to EPA source test techniques, and works with outside groups on test development programs. Source Characterization Group B The Source Characterization Group B, under the direction of the Group Chief, is responsible for a broad range of source characterization activities including emissions testing, methods development, and reviewing emissions methods submitted by other groups. The Group carries out these activities to support the development of national emission standards and implementation of national regulatory programs. It is also responsible for disseminating this and other information to the regulatory and industrial community. This Group works closely with Source Characterization Group A to allocate responsibility for specific source characterization projects. Specifically, the Group: Plans and conduct field test programs to provide quality data in support of regulatory development consistent with data quality objectives Evaluates and reviews test results and methods developed by other groups (both EPA, State/local agencies as well as the private sector), including the review of methods submitted under Method 301 Provides expert evaluation and reports of source characterization data and related documentation in support of regulatory development programs Promotes consistency in the application and use of all source characterization techniques (including emission factors and emission test procedures) by providing expert technical assistance, including reviewing modifications and alternatives to EPA factors and source test techniques, and works with outside groups on test and factor development programs. Provides leadership in emissions testing through preparation and dissemination of technical guidance and other informational documents, test methods and procedures, and test report and emission data summaries through the Emissions Measurement Technical Information Center electronic bulletin board, training workshops, and other media. Groups under the Emmission Standards Division Metals Group The Metals Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 6 of 17 industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc., including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Minerals and Inorganic Chemicals Group The Minerals and Inorganic Chemicals Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc.,including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Combustion Group The Combustion Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 7 of 17 pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc.,including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Coatings and Consumer Products Group The Coatings and Consumer Products Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc.,including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 8 of 17 Organic Chemicals Group The Organic Chemicals Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc.,including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Waste and Chemical Processes Group The Waste and Chemical Processes Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc.,including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 9 of 17 members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Policy, Planning and Standards Group The Policy, Planning and Standards Group, under the supervision of the Group Chief, is responsible for developing and implementing national emission standards, new source performance standards, control techniques guidelines, regulatory review programs, and other technical documents for specific categories of stationary sources of hazardous and criteria air pollutants. The Group performs comprehensive analyses of hazardous and criteria air pollutant emissions and control measures for the specified categories of stationary sources. Such analyses typically form the basis for national emission standards or technical guidance documents. Finally, the Group facilitates planning and development of Division activities and integration of Division programs with other OAQPS and EPA programs. Specifically, the Group: Conducts preliminary assessments of its source categories (number of sources, pollutants, and emission processes) Gathers information on emissions and effectiveness of control measures from State and local agencies, industries, and vendors Estimates control costs and analyzes multi-media environmental impacts of different control alternatives and provides profiles of industry structure and costs for economic analysis Analyzes specific control measures and their effectiveness, costs, operations, etc., including the potential for co-control of hazardous and criteria air pollutants Monitors field tests of representative facilities and assesses applicability of results to other facilities to derive emission factors and recommend achievable performance levels of control measures Develops preambles, regulations, and supporting documentation in conjunction with other work group members and responds to questions and comments both within and outside EPA Provides technical assistance on specific industries, rules, or guidance documents to State, local, and international air pollution control agencies, other EPA offices, and the public Develops or participates in development of "cross-cutting" programs (e.g. general provisions for national emission standards for hazardous air pollutants) which apply to all source categories Participates with other OAQPS elements to identify research and development needs regarding the control of hazardous and criteria air pollutants for source categories of national interest Develops overall plans, strategies, and policies addressing regulatory programs for stationary sources of air toxics and criteria pollutants (NESHAPs, NSPS) Develops new, innovative, and streamlined approaches to regulatory development, including coordinated strategies for co-control of hazardous and criteria air pollutants, and works with other groups, divisions, EPA offices, State and local organizations and industry to implement these approaches Leads Division efforts in coordination with other Divisions to integrate other aspects of the air toxics program requirements into regulatory options (e.g., urban air toxics, residual risk) Leads Division efforts to coordinate overall program regulatory plans with progam budget and contract requirements Integrated Policy and Strategies Group The Integrated Policy and Strategies Group, under the direction of the Group Chief, develops air quality management strategies, policies and strategic measures of success for particulate matter (PM-10), sulfur dioxide ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 10 of 17 (S02), lead, and carbon monoxide (CO) programs. It also manages the implementation on a national level of these specific criteria air pollutant programs. The Group provides technical assistance to Regional Offices and State/local agencies to implement these criteria air pollutant programs at their levels. It monitors the effectiveness of the air quality management process as it pertains to these pollutants, and participates in assessing the need for ongoing changes to the process. The Group targets strategies and policies when the nature of the air quality problem is not well suited to a uniform national program. The Group develops policies to incorporate and encourage cross-program integration of implementation strategies (including co-control of hazardous and criteria air pollutants), the use of innovative regulatory strategies and pollution prevention, and consideration of environmental justice issues. Specifically, the Group: Develops and supports the implementation of national and targeted strategies and policies, including innovative approaches (e.g., market-based, pollution prevention, etc.), for achieving and demonstrating attainment with new and revised NAAQS for PM-10, SO 2, lead, and CO Working closely with other OAQPS Divisions and Groups, develops strategies for co-control of hazardous and criteria air pollutants Develops innovative approaches, policies and regulations, through general preambles and/or Part 51 rules, for new and revised NAAQS for PM-10, SO 2, lead, and CO Coordinates policy development with other EPA offices and Federal agencies, and works closely with State/local agencies and other key interested stakeholders in developing policies. Provides leadership and technical assistance to Regional Offices and State and local agencies on PM-10, S02, and lead implementation programs with regard to national issues such as failures, findings, and sanctions Develops program specific guidance (PSG) for PM-10, SO 2, lead, and CO programs Reviews and analyzes Memorandums of Agreement for Title I ( PM-10, SO 2, lead, and CO) to evaluate strategic direction of Regional programs and fulfillment of priorities articulated in the PSG Ozone Policy and Strategies Group The Ozone Policy and Strategies Group, under the direction of the Group Chief, develops air quality management strategies, policies and strategic measures of success for ozone and nitrogen oxides (NOx). It also manages the implementation on a national level of ozone and nitrogen oxide control programs. The Group provides technical assistance to Regional Offices and State/local agencies to implement these programs at their levels. It monitors the effectiveness of the air quality management process as it pertains to these pollutants, and participates in assessing the need for ongoing changes to the process. The Group develops geographically focused (i.e., Regional, multistate, and local) and targeted strategies and policies when the nature of the air quality problem is not well suited to a uniform national program. The Group develops policies to incorporate and encourage cross-program integration of implementation strategies, the use of innovative regulatory strategies and pollution prevention, and consideration of environmental justice issues. Specifically, the Group: Develops and supports the implementation of national and targeted strategies and policies, including innovative approaches (e.g., market-based, pollution prevention, etc.), for achieving and demonstrating attainment with new and revised NAAQS for ozone and NOx Develops innovative approaches, policies, and regulations, through general preambles and/or Part 51 rules, for new and revised NAAQS for ozone and NOx Coordinates policy development with other EPA offices and Federal agencies, and works closely with State/local agencies and other key interested stakeholders in developing policies. Provides leadership and technical assistance to Regional Offices and State and local agencies on ozone and NOx implementation programs with regard to national issues such as failures, findings, and sanctions Develops program specific guidance (PSG) for Title I (ozone and NOx) programs Reviews and analyzes Memorandums of Agreement for Title I (ozone and NOx) to evaluate strategic ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 11 of 17 direction of Regional programs and fulfillment of priorities articulated in the PSG Manages national elements of the State Implementation Plan (SIP) process, including maintaining the SIPTRAX database, responding to OMB and Congressional inquiries on national issues, and establishing policies and guidance to clarify the relationship between SIP's and operating permits Health Effects and Standards Group The Health Effects and Standards Group, under the direction of the Group Chief, maintains expertise on the health effects of air pollution, conducts assessments of health effects associated with exposure to criteria air pollutants and air toxics, as well as of the nature and extent of such exposure and the resulting risks. These assessments are the health basis for the Group's review and revision of the NAAQS, as well as for its characterization of the public health impacts of OAQPS regulatory programs. The Group participates in relevant Agency research planning in support of the NAAQS and hazardous air pollutant programs, coordinating with ORD. Specifically, the Group: Maintains and provides expertise on health effects associated with exposure to criteria air pollutants and air toxics, for both internal strategy and standards development and external communications (e.g., court testimony, public forums) Reviews, revises and establishes the NAAQS: Manages the interoffice NAAQS review process, including coordination with ORD's Criteria Document reviews and, when appropriate, with OAQPS's implementation strategy and regulatory impacts development Provides expertise on the health effects associated with the health-based primary NAAQS Prepares Staff Papers which integrates science and policy considerations and staff recommendations on the NAAQS, in coordination with the Risk and Exposure Assessment and Visibility and Ecosystem Protection Groups. Prepares associated proposal and promulgation regulatory packages Risk and Exposure Assessment Group The Risk and Exposure Assessment Group, under the direction of the Group Chief, maintains the scientific and analytical expertise necessary to conduct current and develop new assessment methodologies and policies for human exposure, health risk, and risk communication, and to serve as a center of health risk information for Regional, State, and local agencies. It participates in Agency research planning in support of the NAAQS and hazardous air pollutant programs, coordinating with ORD, and in the development of benefits assessment methodologies by the Economics and Innovative Strategies Group. Specifically, the Group: Maintains and provides expertise on human exposure and health risk assessment for criteria air pollutants and air toxics, and on risk communication, for both internal strategy development and external communications (e.g., operation of the Air Risk Information Center, court testimony, public forums) Conducts human exposure and health risk assessments in support of the hazardous air pollutant program and the development of Maximum Achievable Control Technology (MACT) standards, including Evaluating petitions to add and delete substances from the list of hazardous air pollutants Evaluating substitutes for hazardous air pollutants Assessing health effects and health risks associated with setting floors for MACT standards Assessing area sources and makes findings on source category deletions Assessing averaging and trading options for MACT standards Conducting residual risk determinations ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 12 of 17 Conducts human exposure and health risk assessments in support of NAAQS reviews, and other regulatory and programmatic impact assessments Develops strategies and participates in Agency-wide studies and policy and regulatory development involving human exposure and health risk assessment (e.g., response to NAS risk assessment study) Visibility and Ecosystem Protection Group The Visibility and Ecosystem Protection Group, under the direction of the Group Chief, conducts assessments of ecological effects associated with exposure to air toxics and criteria air pollutants, develops strategies for ecologically and geographically-based environmental and air quality management problems, and maintains expertise on the ecological and welfare effects of air pollution. It also manages Office of Air and Radiation's (OAR) visibility protection program, including developing rules and guidance and providing leadership for geographically-based visibility programs. The Group participates in relevant Agency research planning in coordination with ORD, and in the development of benefits assessment methodologies by the Economics and Innovative Strategies Group. Specifically, the Group: Develops, maintains and provides expertise on ecological and other welfare effects associated with criteria air pollutants and air toxics, for both internal strategy and standards development and external communications Participates in the NAAQS review process by providing expertise on ecological and other welfare (e.g., visibility impairment, materials damage) effects associated with the welfare-based secondary NAAQS and prepares the sections of staff papers which integrate science and policy issues and provides staff recommendations on secondary NAAQS Develops strategies and policies to facilitate Regional/State design of appropriate geographically focused and crosscutting initiatives (e.g., urban air toxics) and provides technical support to the implementation of geographically focused and crosscutting (air toxics and criteria air pollutant) programs Conducts source identification and the characterization of relative ecosystem loadings and air pollutant effects to assist in the development of strategies for ecological and geographically focused air quality and environmental management problems (e.g., the Great Waters Program, visibility protection, Urban Air Toxics Program) Conducts studies of air pollutants and source categories related to the development of integrated and geographically focused strategies (e.g., Mercury study, Utility study, listings required by Section 112(c)(6)) Conducts assessments of ecological effects in support of NAAQS reviews and other regulatory and programmatic impact assessments Manages visibility protection program, including developing strategies for national and geographically focused programs, coordinating efforts with other air quality management programs, promulgating rules, issuing policies and guidance, and providing leadership to Regions and State and local agencies on visibility protection issues Coordinates visibility protection efforts with other Federal agencies and selected organizations (e.g., Department of the Interior, Grand Canyon Visibility Commission, Southeast Air Management Initiative, etc.) to ensure effective development and implementation of programs Innovative Strategies and Economics Group The Innovative Strategies and Economics Group, under the direction of the Group Chief, develops methodologies and conducts analyses of costs, benefits, and economic and regulatory impacts of air quality management strategies, programs, and regulations developed throughout OAQPS. The Group maintains the expertise to develop and apply new methodologies for benefits assessments and participates in relevant Agency research planning in coordination with ORD and the Office of Policy, Planning and Evaluation (OPPE). It also participates in Agency-wide assessments of economic impacts and benefits of air programs. The Group also ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 13 of 17 manages OAQPS's Innovative Regulatory Strategies Program through an interoffice planning and coordination team, planning, prioritizing, and coordinating ongoing and new activities (e.g., coordination of State grants for economic incentives). Specifically, the Group: Performs and documents a wide range of economic analyses, such as economic and regulatory impacts (including costs and benefits), regulatory flexibility (e.g., impacts on small entities), information collection requests, and environmental justice, primarily in support of the NAAQS program and strategies for geographically focused and crosscutting problems (e.g., visibility impairment, urban air toxics), as well as the regulatory development of national emission standards required under Clean Air Act Titles I and III Presents the results of such analyses to the Office of Management and Budget and other Federal agencies and the public, when necessary, to facilitate external program reviews Develops methodologies for and provides expertise on benefits analyses, coordinating with the Risk and Exposure Assessment Group, the Visibility and Ecosystem Protection Group, and other EPA offices in support of the application of such methods Provides technical support to Regional, State, and local agencies, international agencies, and the public on economic impacts of criteria air pollutant and air toxics programs and on benefits methods and analyses Coordinates OAQPS's Innovative Regulatory Strategies Program, including coordinating the development of rules, policies, and guidance on market-based incentive programs in support of policy and regulatory development across OAQPS Provides leadership and coordinates technical support to Regional, State, and local agencies, other Federal agencies, international agencies, and the public on economic and market-based incentives, partly through participation in program design and implementation initiatives. Groups under the Information Transfer and Program Integration Division Operating Permits Group The Operating Permits Group (OPG), under the direction of the Group Chief, is responsible for developing national regulations and guidance as required under the Title V operating permits program and SIP operating permit programs and for providing technical assistance on program implementation to EPA Regional Offices and State and local agencies. Specifically, the Group: Develops and modifies EPA regulations dealing with operating permits as required under Title V of the CAAA. Develops policies and guidance on Title V and SIP operating permit programs, coordinating with other EPA programs that will use operating permits to implement their program. Establishes and maintains cooperative working relationships with Regional Offices, State and local agencies, and interested parties outside the government on regulatory and policy issues related to the operating permits program. Develops support material to assist Regional Offices in the review and approval/disapproval process of State-submitted operating permit programs and permits. Reviews Regional Office recommendations for approval/disapproval of State programs and permits. Determines and develops technical guidance and other support to assist Regional Offices and States in implementing operating permit programs. In conjunction with the Education and Outreach Group, determines training needs and develops appropriate training materials. Establishes national operating permit program information requirements and other performance measures that can be used to evaluate the program. Coordinates resolution of selected technical support issues with national implications with Regional Offices and State and local agencies. ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 14 of 17 Develops audit guidance for State operating permit programs in conjunction with the Program Review Group. Supports Regions in conducting audits of State operating permit programs. Serves as national program manager for the operating permits program, including determination of the status of the program. Provides input to the development of the annual Program Specific Guidance and supports the implementation of the MOA process with respect to operating permit programs. Integrated Implementation Group The Integrated Implementation Group, under the supervision of the Group Chief, manages the development and implementation of requirements under the new source review (NSR) and prevention of significant deterioration (PSD) provisions of the Clean Air Act. The Group also manages the national implementation, working closely with Regional Offices and State/local agencies, of air toxics program requirements under Section 112 of the Clean Air Act (including sections 112 (g), 112(1), 112(j), and 112(r)). In addition, the Group promotes integration of NSR/PSD, operating permits, air toxics, and related programs. Specifically, the Group: Develops and modifies EPA regulations dealing with NSR and PSD. Develops and modifies policies and regulations involving strategies to reduce public exposure to toxic air pollutants (112(g), 112(1), 1120, and 112(r) among others). Develops and issues national policy and guidance on NSR/PSD and the implementation of air toxics programs, including guidance on the integration of air toxics programs with operating permit programs. Provides technical assistance to and coordinates with EPA Regional Offices, State and local agencies, and interested parties outside government on NSR/PSD and air toxics implementation programs. Manages the integration of air toxics rules with the operating permit and NSR/PSD programs, and coordinates the identification and resolution of cross-program implementation issues. Continuously assesses NSR/PSD and air toxics program operations and processes and develops and implements improvements. Establishes measures of success and serves as the focal point for measuring the effectiveness of NSR/PSD and air toxics program implementation. Develops program specific guidance (PSG) for NSR/PSD and air toxics programs. Reviews and analyzes Memorandums of Agreement for NSR/PSD and air toxics programs in order to evaluate strategic direction of Regional programs and fulfillment of priorities articulated in the PSG. Information Management Group The Information Management Group, under the supervision of the Group Chief, manages the design, operation, maintenance, and continual improvement of wide-scale data bases and information system platforms which assist the development and implementation of air programs. The Group assesses current information system technologies and works closely with the Information Transfer Group and other OAQPS program groups to define and develop new program information management tools and strategies to continually improve information management systems to meet customer and user needs. Customers and users include: OAQPS, EPA Headquarters and Regional Offices, State and local agencies, industry, consultants, environmental groups, and the international enviromental community. Specifically, the Group: Provides leadership to EPA Regional and State/local agency user communities on data management issues. In cooperation with customers, user communities, and the National Data Processing Division (NDPD), identifies emerging data management needs and plans and designs information management systems and tools that efficiently and effectively meet user/client needs. ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 15 of 17 Designs, operates, maintains, and continually evaluates the effectiveness of wide-scale information systems for the collection, storage and retrieval of air quality, emissions and compliance data (e.g., The Aerometric Information Retrieval System (AIRS), its subsystems, and other systems under development (MACT data base, permits data base)). Develops and establishes compatibility guidance and standards for use by State/local agencies with independent data management systems to facilitate data transfer and conversion. In coordination with the Information Transfer Group, provides guidance and direction on the enhancement of existing information management and transfer system platforms and participates in the design and implementation of new system platforms that are compatible with current and emerging program and user needs. Works with the Office of Information Resources Management and the National Data Processing Division to ensure that all tools and systems meet applicable Federal and Agency standards for data management. In coordination with the Information Transfer Group provides technical assistance to Regional and State/local users on data entry, data retrieval, coding of data, reporting, searches, and related questions; works with the Education and Outreach Group on outreach and training on systems and tools. In cooperation with national program managers and Regional, State and local agencies, ensures smooth flow of air pollution control data and information in the OAQPS systems. Information Transfer Group The Information Transfer Group, under the supervision of the Group Chief, is responsible for insuring broad public access to policies, regulations, guidance, data bases and other information generated by EPA's air programs. The Group coordinates with responsible offices on the timely release of air related information products. Develops new tools and applications that enhance access to and usefulness of the data and information residing in OAQPS information management systems. Users and clients include, as appropriate: EPA Headquarters and Regional Offices, State and local Agencies, industry, consultants, environmental groups, the general public, and the international environmental community. The Group works closely with the Information Management Group to evaluate current and future information transfer needs, supports the design of new information management systems, and manages information transfer activities (e.g. the Technology Transfer Network, (TTN), Control Technology Center (CTC), RACT/BACT/LAER Clearinghouse, National Air Toxics Information Clearinghouse (NATICH)). In addition, the Group continually evaluates the degree of use of information and management systems to assess customer satisfaction and future information transfer needs. Specifically, the Group: Provides leadership to OAQPS, EPA Regional Offices, and the State/local agency user community on information transfer issues. Provides leadership and coordination to information transfer staff (e.g. TTN sysops, technical support centers) in each OAQPS group and other OAR and EPA offices concerned with air related information products. Works closely with the Information Management Group in improving the transfer capability of OAQPS information management systems. Works closely with the Information Management Group and the Education and Outreach Group and assists in developing and conducting training in new information management and transfer tools. In cooperation with the user community and EPA information providers, continually assesses the quality, convenience, and usefulness of information transfer capabilities and develops and implements enhancements that improve access and increase the usefulness of transferred information (e.g. AIRS Graphics, AIRS Executive). Operates and maintains information transfer activities(e.g., the TTN, the CTC, the RACT/BACT/LAER Clearinghouse (RBLC), NATICH) and ensures that information transfer systems resident in other OAQPS Groups are operated and maintained. Through the CTC, provides access to EPA technical expertise and information, and provides substantive ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 16 of 17 technical assistance and develops and provides products and tools (e.g., direct engineering assistance, technical guidance documents, evaluation of emerging technology, and software tools) to State and local agencies and, on a cost recovery basis, to others on the application of control technology and pollution prevention methods. Provides assistance to States in developing and implementing the small business stationary source technical and environmental assistance program and coordinates small business support activiites with other OAQPS information transfer components, the Pollution Prevention Division, the Chemical Emergency Preparedness and Prevention Office, and the EPA Small Business Ombudsman. Education and Outreach Group The Education and Outreach Group, under the supervision of the Group Chief, is the OAQPS focal point for air pollution education and training. The Group operates the Air Pollution Training Institute and the associated Distance Learning Network, providing technical courses, conferences and workshops to all levels of the air program. The Group also provides leadership and expertise to develop and distribute educational materials which improve public understanding of air pollution issues and effectively communicate the air program's mission and goals. Specifically, the Group: Manages training services for EPA, State and local air pollution control programs, including operating the Distance Learning Network, the Air Pollution Training Institute, and training services through associated universities. Manages the development of educational course curricula and materials used in courses and evaluates performance in teaching methods, reference materials, and overall effectiveness of presentations. In cooperation with the Regional, State, and local agencies and others in the air program community (e.g., the Air and Waste Management Association), identifies training needs and trends and continuously improves education and training efforts in order to meet client needs. Provides expertise to all OAQPS elements in developing and presenting special purpose workshops and meetings and in improving both the quality of the information and the methods of delivery. Administers training grants and special fellowship programs Working with the program and information groups, provides leadership to identify the appropriate audiences for air pollution program information, assesses the level of information required for effective communication, determines the best method for delivering the information to the target audience, and coordinates delivery. Provides leadership and expertise to develop and distribute air pollution educational materials used to improve public understanding and to assist in the implementation of environmental protection programs. Program Review Group The Program Review Group, under the supervision of the Group Chief, manages the review of major air programs (e.g., air toxics implementation, ozone program, operating permits program, etc.) that cut across Division, Office, and Agency lines. The Group establishes criteria for evaluating programs, and works closely with other OAQPS Groups and programs, as well as other Headquarters, Regional, State/local and other organizations involved to determine the scope of reviews as well as benchmarks and measures of progress. The Group works closely with all levels of the air program to assess program strengths and weaknesses and to assist in the development and implementation of program improvements. Specifically, the Group: Develops criteria and methods for program review. In cooperation with affected programs, identifies scope and timing of program reviews and develops program review plans. In coordination with affected programs and information groups, compiles reports and other information necessary for program reviews. ------- OAQPS Appendix A Revision: 0 Date: 6/96 Page 17 of 17 Works with other OAQPS Groups and programs to determine measures of performance and progress (e.g., environmental indicators, MAPS, budget, etc.) against which to evaluate programs. Works with identified organizational elements and levels of the air program to carry out program reviews and to identify and implement improvements. Serves as focal point of expertise for other OAQPS Groups and programs on methods and processes for assessing program performance and developing improvements (e.g., continuous improvement methodology, performance measures, etc.). Assists the Immediate Office of the Director, OAQPS in serving as a focal point for Congressional, General Accounting Office, and Inspector General inquiries. ------- OAQPS Appendix B Date: 6/96 Page 1 of 9 Appendix B Contract QA Form and Guidance The following form was modified for use by OAQPS personnel when developing contracts, assistance agreements, and interagency agreements. The form shall be used on any funding package in which there are environmental data operations. It must also be developed for any funding activity over the small purchase threshold. The guidance provides a rational for the questions and will help personnel complete the form ------- Quality Assurance Review for Extramural Projects (Contracts/Work Assignments/Assistance Agreements/IAGs) I. GENERAL INFORMATION Descriptive Title: ID# Sponsoring Program Office : Duration: Funding Mechanism C fWA/Al I II. This Specific Action Requires Environmental Data Operations? Circle If yes, complete remainder of Form with the OAQPS QA Officer If no, sign and submit Form and Scope of Work to the OAQPS QA Officer YES NO III. Quality Assurance Requirements: (projects requiring environmental data operations) Circle Only answer questions specific to the appropriate funding mechanism as indicated in the "FundingMech. " column (C-Contract, WA- Work Assignment, A- Assistance Agreement, I-IAGs) Yes No Funding Mech. Submission of a written quality management plan (commitment of the offeror's management to meet the QA requirements of the scope of work) is to be included in the contract proposal or work plan. Yes No C Submission of a written QA project plan is to be included in the proposal or work plan Yes No C/WA/A/I A written QA project plan is required as part of the contract/work assignment/project Yes No C/WA/A/I Performance on available audit samples or devices shall be required as part of the evaluation criteria Yes No C An on-site evaluation of proposer's facilities will be made to ensure that a QA system is operational and exhibits the capability for successful completion of this project - Technical systems audits are required for pre-award evaluation Yes No c Yes No c - Technical systems audits are required during the project period Yes No C/WA/A/I QA reports will be required (either progress or final reports) Yes No C/WA/A IV. Determination (projects requiring environmental measurements) Percentage of technical evaluation points assigned to QA for contract award (contracts only) WAM/Project Manager estimate of percentage of costs allocated to environmental measurements V. The signatures below verify that the QA requirements have been established. QA Officer Date WAM/Project Manager Date A signed copy of this form must be included with the Request for ProposaUcontract). Work Assignment. or Proposal (assistance agreementZIAGi when sent to the Contracts Office/Grants Management. A copy must be placed on file with the Project Officer/Work Assignment Manager Note: If this specific action requires a QA project plan, be sure this information is included in the funding conditions ------- EXCERPT FROM EPA 1900 CONTRACTS MANAGEMENT MANUAL SECTIONS 2.4 AND 2.5 2.4 APPROVALS All procurement request packages must contain required approvals before the CO may process the action (see Figure 2-2). 2.5 QUALITY ASSURANCE (QA) REQUIREMENTS This section sets forth policies and procedures concerning QA requirements for contracts in excess of the FAR threshold for "small purchases. " a. Requirement for the QA Review Form. (1) The QA Review Form (Figure 2-3) shall be completed as required below to assure that all environmentally-related measurements which are funded by EPA or which generate data mandated by EPA are scientifically valid, defensible, and of known precision and accuracy. (2) Each Procurement Request/Order (EPA Form 1900-8), except for incremental funding actions, will be accompanied by a QA Review Form (Figure 2-3) when the acquisition is in excess of the FAR threshold for "small purchases" and is in one of the following cost categories as set forth in the accounting and object classification structure: (a) 25.32 Programmatic Research and Development Contracts (b) 25.05 Program Contracts (c) 25.63 Programmatic Occupational Health Monitoring (d) 26.29 Programmatic Scientific and Technical Laboratory Supplies (e) 31.01 Programmatic Scientific and Technical Equipment b. Responsibilities. The PO will obtain a signed and completed QA Review Form from the QA Officer responsible for monitoring the PO's program. If the QA Officer indicates on the form that QA requirements are applicable to a procurement and the potential value of the procurement exceeds $500,000, the QA Officer, or designee, shall be a member of the Technical Evaluation Panel to evaluate QA aspects of the technical proposals. Program Offices will establish procedures for QA Officer review of proposals on procurements of S500.000 or less. NOTE: Figure 2-3, which is referenced in the following text is the form on the opposite side. It has been reformatted to fit on a single page. TERMS AND DEFINITIONS Definitions derived from ANSI/ASQC E4-1995 American National Standard -Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs. Environmental data operation - work performed to obtain, use, or report information pertaining to environmental processes and conditions. Environmental data- any measurements or information that describe environmental processes or conditions, or the performance of environmental technology. Environmental process- manufactured or natural processes that produce discharges to or that impact the ambient environment. Environmental conditions- the description of a physical medium (e.g. air, water, soil, sediment) or biological system expressed in terms of its physical, chemical, radiological, or biological characteristics. QA project plan (QAPP)- A formal document describing in comprehensive detail the necessary QA/ QC, and other technical activities that must be implemented to ensure the results of the work performed will satisfy the stated performance criteria Quality management plan (QMP)- a formal document or manual, usually prepared once for an organization, that describes the quality system in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing activities conducted. REGULATIONS AND GUIDANCE EPA QA/R-1: EPA Quality Systems Requirements for Environmental Programs EPA QA/R-2: EPA Requirements for Quality Management Plans EPA QA/R-5: EPA Requirements for Quality Assurance Project Plans EPA QA/G-5: Guidance on Quality Assurance Project Plans EPA QA/G-6: Guidance on Standard Operating Procedures for Environmental Data Operations EPA QA/G-8: Guidance on Technical assessments for Environmental Data Operations EPA QA/G-9: Guidance for the Data Quality Assessment Process ANSI/ASQC E4-1995 American National Standard -Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs ------- Date: 10/96 Page 4 Guidance For the Use of The Quality Assurance Review Form For Extramural Projects VC^ OAOPS Office of Air Quality Planning and Standards --¦ 6-v and stgS^United States Environmental Protection Agency ------- Date: 10/96 Page 5 Guidance for use of the Quality Assurance Review for Extramural Projects Introduction The OAQPS QA Review Form is a modification of Figure 2-3 in the Contracts Management Manual (EPA 1900). It has been revised to encompass work assignments, assistance agreements, and interagency agreements (IAGs). For the remainder of this document, the term "project" will be used to describe these various funding mechanisms. The intent of the QA Review Form is to: ~ Document that both the QA Officer and the person responsible for the technical aspects of the contract/assistance agreement have reviewed the project and concur that: 1) the project does/does not contain environmental data operations (EDOs), and 2) if it does contain EDOs, that appropriate QA activities will be implemented. A discussion on EDOs are provided below. ~ Provide information for the tracking of all OAQPS EDOs. ~ Provide the Work Assignment Manager/ Project Manager, with a list of questions to decide whether various QA activities will be required in the project that would then need to be specified in the procurement package, work assignment, or assistance agreement special conditions. ~ Provide the Contracting Officer/Project Officer with information on QA requirements that need to be included in the procurement package, work assignment or assistance agreement and to confirm adherence to the terms of the contract. Environmental data operations (Programs) EPA has developed a revised QA policy and has adopted the American National Standard- Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs. This is also called ANSI/ASQC E4, or E4 for short. The following definitions from E4 will help to explain the term EDO. Environmental data operation - work performed to obtain, use, or report information pertaining to environmental processes and conditions. Environmental Programs - any work or activities involving the environment including: characterization of environmental processes and conditions environmental monitoring; environmental research and development; the design, construction and operation of environmental technologies; and laboratory operations on environmental samples. Environmental data - any measurements or information that describe environmental processes or conditions, or the performance of environmental technology. Environmental process - manufactured or natural processes that produce discharges to or that impact the ambient environment. Environmental conditions - the description of a physical medium (e.g. air, water, soil, sediment) or biological system expressed in terms of its physical, chemical, radiological, or biological characteristics. By combining the terms environmental programs and environmental data one can get a good sense of an EDO. In summary, an EDO is any project that includes the acquisition and/or interpretation of environmental data. However, by this statement, and included in E4 is the idea that projects using secondary data . meaning that the project is not acquiring new environmental data, is considered an EDO and will require some degree of QA. Since EPA has not developed a policy, specific guidance, or regulations on the implementation of secondary data QA processes, OAQPS, at present, expresses the following statement: Although the Agency currently recognizes that there should be some level of QA associated with secondary data, there is presently no policy, regulation, or guidance to address this issue. Henceforth, there are presently no OAQPS requirements for QA of secondary data . Therefore, when a QA Form for a project containing secondary data is submitted to a QA Officer, the QA Officer will identify the project as an EDO (see Section II) and will place a label on the form that includes the above statement. It is suggested that the person responsible for the technical aspects of the project (e.g., WAM ) will work with the contractor to develop some manner of assessment of the quality of secondary data used for the project. To summarize, it is EPA policy that all EPA funded EDOs require the development of a QA project plan. Therefore, a QA Review Form should accompany any acquisition or assistance agreement that includes EDOs. As indicated in the ------- Date: 10/96 Page 6 Contracts Manual, any procurement greater than the small purchase threshold requires the submission of a QA Review Form, regardless of whether EDOs are included in the project. The intent of this document is to provide additional information to the person attempting to complete this form. It can be used to improve the decision making process, as well as the quality of the data derived from projects containing EDOs. Section I I. GENERAL INFORMATION Descriptive Title: ID # Sponsoring Program Office: Duration: Funding Mechanism C/WA/A/I This section is used simply to provide information on the procurement for adminstrative purposes. If the QA Form was misplaced, this information would allow one to find the correct file. ~ Descriptive title -Official title used to describe the overall project ~ ID#- This is the unique identifier assigned by Contracts or Grants Office for this project. This may/may not be available at the time of submission but is the best piece of information for tracking purposes. ~ Sponsoring Program Office- Funding office (e.g., OAQPS, EMAD, MQAG). It would be appropriate to identify down to the group level. ~ Duration- Length of project ~ Funding Mechanism- C= Contract, WA = Work Assignment, A= Assistance Agreement, I = IAG Section II II. This Specific Action Requires Environmental Data Operations? Circle YES NO If yes, complete remainder ofform and submit signed copy ofform to OAQPS QA Officer If no, sign and submit form and Scope/Statement of Work to the OAQPS QA Officer This question is used to determine whether the project includes any EDOs. EDOs are the process by which information is collected on environmental conditions, processes, or the performance of environmental technology. As indicated on the Form, by circling YES, the remainder of the Form must be completed. By circling NO, the WAM/Project Manager, as the Grant/Contracting Offices technical representative, is indicating that no environmental data operations will occur in this project. The Form would be included in the funding package and be submitted to the QA Officer (QAO). The QAO has the independent responsibility to ensure that the project does not include EDOs and must therefore review the work assignment. Note: At times, activities within a project change and there is a possibility that within the project period, environmental data operations will be added. It becomes the responsibility of the WAM/Project Manager to adhere to EPA QA requirements. ------- Date: 10/96 Page 7 Section III III. Quality Assurance Requirements: (projects requiring environmental data operations) Circle Only answer questions specific to the appropriate funding mechanism as indicated in the "Funding Mech." column (C-Contract, WA-Work Assignment, A-Assistance Agreement, I-IAGs) Mech Submission of a written quality management plan (commitment of the offeror's management to meet the QA requirements of the scope of work) is to be included in the contract proposal or work plan. Yes No C Submission of a written QA project plan is to be included in the proposal or work plan Yes No C/WA/A/I A written QA project plan is required as part of the contract/work assignment/project Yes No C/WA/A/I Performance on available audit samples or devices shall be required as part of the evaluation criteria Yes No C An on-site evaluation of proposer's facilities will be made to ensure that a QA system is operational and exhibits the capability for successful completion of this project - Technical systems audits are required for pre-award evaluation - Technical systems audits are required during the project period Yes No c Yes No c Yes No C/WA/A/I QA reports will be required (either progress or final reports) Yes No C/WA/A Submission of a written quality management plan (commitment of the offeror's management to meet the QA requirements of the scope of work) is to be included in the contract proposal or work plan (Contracts Only) The quality management plan, previously known as the QA program plan is defined as: " a formal document or manual, usually prepared once for an organization, that describes the quality system in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing activities conducted' Many environmental contractors have developed this type of document. The criteria for the development of a QMP is included in " EPA QA/R-2: EPA Requirements for Quality Management Plans" This question refers to whether a quality management plan will be submitted along with the contract proposal. It is advantageous to review this during the technical evaluation phase of a procurement to ascertain the emphasis the proposal places on quality. However, the procurement package as well as the RFP must indicate that the submission of a QMP is required. Submission of a written QA project plan is to be included in the proposal or work plan (Contracts, Work Assignments, Assistance Agreements, IAGs) The QA project plan is the formal document, describing in comprehensive detail, the necessary QA/QC and other technical activities that must be implemented to ensure the results of the work performed will satisfy the stated performance criteria. The QA project plan is specific to the EDOs of the proposed project. As stated earlier, QA project plans are required for every EDO. The criteria for the development of a QA project plan is included in "EPA QA/R-5: EPA Requirements for Quality Assurance Project Plans" and "EPA QA/G-5: Guidance on Quality Assurance Project Plans" This question determines whether the QA project plan will be submitted along with the proposal/work plan, or will be allowed to be submitted once the funding for the project is awarded. Depending upon the extent to which EDOs are part of the project, a contractor/grantee could spend a considerable amount of resources developing a project specific QA project plan for a project that may not be awarded to that contrac tor/grantee. However, it is a mechanism that can be used and is advantageous in the evaluation process. Oftentimes, to avoid unnecessary costs to be incurred by the contractor, a preliminary QA project plan, which broadly outline the QA/QC activities that must be implemented, is submitted with the ------- Date: 10/96 Page 8 proposal/work plan with a detailed plan requested prior to the commencement of the work approved by the government. It should be noted that grants are more flexible than contracts. Grants usually go through some type of pre-proposal stage where they are evaluated and selected for potential funding (thereby reducing the field and providing some assurance that the grant will be funded). Once the selections are made, the grantees are then asked for full proposals. At this stage, a QA project plan could be solicited. New grant regulations (40 CFR parts 30 and 31) have been developed to allow for some expenditure of funds prior to award of grants. These funds could be used for the development of a QA project plan. However, these regulations should be confirmed with the grants office. Performance on available audit samples or devices shall be required as part of the evaluation criteria (Contracts Only) Contracts go through a technical evaluation panel (TEP) process which judge a number of aspects of the potential contractors. One aspect of the evaluation is the determination of acceptable performance of the data collection activities. Reference samples are provided to each potential contractor and are scored. Depending upon how the Scope of Work is developed, this score is included in the overall scoring of the contractor. A device could mean a number of things and it is important to include this. Some contractors bid on work without having the actual measurement device (sampling/analytical equipment) in house. This action should ensure that the technical evaluation will include an assessment of the devices used in actual project implementation. An on-site evaluation ofproposer's facilities will be made to ensure that a QA system is operational and exhibits the capability for successful completion of this project (Contracts Only) The PO is ultimately responsible for ensuring that the project successfully accomplishes the objective. All efforts should be made, prior to award, that the best contractor within the competitive range is awarded the contract. The on-site evaluation is one mechanism that can help to accomplish this. A technical systems audit is usually implemented while the contractor is performing the data collection activity (see section V). Since the contract has not been awarded, the contractor may not be implementing any data collection activities similar to the project. Therefore, an on-site evaluation could be performed with the goal of determining that the contractor has the potential to successfully complete the service. For example, if the contract was for the analysis of 500 samples a month, an on-site evaluation might determine that the lab, with only one analysis instrument, has the potential to fail the sample turnaround requirement. Technical systems audits are requiredfor pre-award evaluation (Contracts only) A technical systems audit is defined as a thorough, systematic, on-site, qualitative audit of facilities, equipment, personnel, training, procedures, record keeping, data validation, data management, and reporting aspects of a system. Information on audits should be included in the procurement package/special conditions. This question establishes whether a technical systems audit will be performed as a pre-award evaluation. This question is somewhat different then the question posed above in that this audit would encompass a detailed review of the actual measurement activity. Technical systems audits are required during the project period (Contract, Work Assignments, Assistance Agreements, IAGs) This question establishes whether technical systems audit will be required during the course of the project period. QA reports will be required (either progress or final reports) (Contracts, Work Assignments, Assistance Agreements, IAGs) In most cases, the answer to this question will be yes, and ensures that all parties (Grants, Contracts, WAM/Project Manager, QAO) are aware of the requirement and include it in the procurement package/work assignment/special grant conditions to ensure potential awardees are cognizant of EPA QA policy . Some organizations designate projects into one of four categories as developed by EPA Risk Reduction Laboratory . Information on these categories can be found in "Preparing Perfect Project Plans (EPA/600/9-89/087)". WAMs/Project Managers may need to consult with the QA Officer when determining the category for a specific data collection activity. QA reports are not be required for category 4 projects, but can be requested by the WAM/Project Manager. ------- Date: 10/96 Page 9 Section IV IV. Determination (projects requiring environmental measurements) Percentage of technical evaluation points assigned to QA for contract award (contracts only) WAM/Project Manager estimate of percentage of costs allocated to environmental measurements % % Percentage of technical evaluation points assigned to OA for contract award (contracts only) During the evaluation phase of contracting, a potential contractor can be judged in a number of areas. One area is QA. This can include the quality of their QA documentation (QA management plan or QA project plan), analysis of reference samples, on-site evaluations, as well as other criteria. A percentage of the overall score for a contractor should include "QA points". The question attempts to find out what percentage will be assigned for QA. The percentage of QA points is usually related to percentage of costs allocated to environmental measurements (i.e., the greater the percentage of resources related to EDOs, the greater the proportion of points allocated to QA). The technical evaluation scoring is included in the Request for Proposal and should allocate at least 5% of the total allocated points to QA. WAM/Project Manager estimate of percentage of costs allocated to environmental measurements The question attempts to get a "ballpark" ( + 15%) estimate of the percentage of the project that will allocated to environmental data operations. This would include all costs associated with sampling, analysis, data reporting/interpretation, and data storage/archive. This information allows one to determine the extent to which : ~ Technical evaluation points would be allotted to QA ~ A QA management plan would be required ~ The category/detail of QA project plan. ~ Funds within the contract are allocated to QA activities. Implementation Process The following figure illustrates the process for completing the QA Form. It is important that the WAM bring the Scope of Work (SOW) or work assignment with the QA Form for the QA Officer to review. The QA Officer is not obliged to review this information immediately, but is required to complete his/her review within 24 hours. WAM QAO 1. Filling out the QA Form 2. Reviews SOW/WA 4. Ensures Form is included retains a copy for flies. 3. Ensures Form is included, checks signatures, retains a copy, and sends Form to CMD ------- Appendix C Revision 0 Date: 5/96 Page 1 of 17 Appendix C QA Project Plan Review Forms The following forms will be used to review all QA project plans in which environmental data operations are the responsibility of GLNPO. These forms are very similar to review forms used in EPA Regions 10 and 3. ------- Appendix C Revision 0 Date: 5/96 Page 2 of 17 Quality Assurance Project Plan Review Document Title: Revision # QAPjP Category # Funding # Author: Project Officer: Date Received: Date Review Requested By: Major deficiences were found in the following elements: _ Title Page _ Table of Contents _ Project Description _ Project Organization _ QA Objectives _ Sampling Procedures _ Sample Custody Calibration Procedures Analytical Procedures Data Reduction Internal QC Checks Audits Preventative Maintenance _ Routine Procedures ... Corrective Action QA Reporting See the attached for discussion comments relative to all elements. Conclusion/Recommendation Acceptable Acceptable with Unacceptable minor revision revise QA Reviewer: Date: ------- Appendix C Revision 0 Date: 5/96 Page 3 of 17 1) Title Page with Provision for Approval Signature & 2) Table of Contents IA iu Nl NA I) Title Paqe Does paqe include: 1) Title of project 2) Name (s) of principal investigators and affiliation 3) Appropriate approval personnel 4) Appropriate QAPjP category II) Table of Contents Does Table include: 1) List of all Plan required elements and appropriate paqe numbers 2) Include list of Appendices Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 4 of 17 3) Project Description IA IU Nl NA Are the following addressed (or referenced), consistently presented, technically correct. 1) Statement of general objectives (purpose) 2) Dates of start and completion of project and sampling activities (schedule 3) Overview of project's scope (activities) 4) Specific objectives for this phase of work 5) Background information 5a Description of site 5b Site history 6) Brief statement of intended data uses *7) Description of sampling network design and rationale *7a Design of overall monitoring system *7b Specific location of sampling sites *7c Justification of overall design *8) Sample matrices *9) Sample locations *10) Parameters to be measured *11) Freguency of collection *12) Field and lab measurements 13) Types of samples 14) Will data meet its intended use Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary * - Information may be included in this section or in appropriate sampling or laboratory procedure sections. The questions apply regardless of location. Comments: ------- Appendix C Revision 0 Date: 5/96 Page 5 of 17 4) Project Organization IA IU Nl NA 1) Does the Plan identify key people for: 1a Overall QA/QC (EPA/extramural) 1b Sampling operations and sampling QC 1c Sample preparation operation and preparation QC 1d Laboratory analyses and laboratory QC 1e Data reviews and data review oversight 1f Performance and systems audits (field, prep, and lab) 2) Are phone numbers and addresses included 3) Is line authority for all referenced organizations explained or demonstrated by including organizational charts 3a Are contractors and subcontractors included in organizational chart 4) Are personnel gualifications included 5) Is the organizational structure appropriate to accomplish the QA objectives of the project Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 6 of 17 5) QA Objectives (DQOs) IA IU Nl NA 1) Is there a statement of intended data usaqe 2) Are the terms and definitions for precision, accuracy, representativeness, comparability, and completeness properly used and expressed. 3) Are DQOs quantitatively stated for precision, accuracy, and completeness 3a Have measurement quality objectives (MQO) been defined for each matrix and parameter and at various measurement phases in order to meet DQO Level of QA effort (frequency of QA) Accuracy (matrix spikes, surrogate spikes, standard solutions, PE samples) Precision (duplicates, replicates, analytical splits) Detection limits (system, method, instrument) Statistical reporting units 3b Are quantitative limits established for each 3c Are objectives presented in table format 3d Is it clear that a distinction has been defined for "total" system variability and bias and not just lookinq at laboratory 4) Are representativeness and comparability appropriately addressed 5) Is there an appropriate amount of QA in the system to control and evaluate the accomplishment of the DQOs. 6) Is the QE/QC sample desiqn presented in a flowchart. Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 7 of 17 6) Sampling Procedure IA iu Nl NA 1) Does the Plan: 1a Provide specific guidance for all field work 1b Provide a mechanism for planning and approving site activities 1c Ensure that sampling activities are limited to those that are necessary anc sufficient 1d Provide a common point of reference for all parties to ensure comparability and compatibility between all activities performed at the site 2) Are the following elements included: 2a Site background 2b Analytes of interest 2c Sample types 2d Map of locations to be sampled 2e Sample locations and freguency 2f Technique or guideline used to select sites 2g Method summary 2h Sample handling and preservation 2i Interferences 2j Safety 2k Eguipment/Material/ Reagents 21 Calibration (see element 7) 2m Procedure 2n Calculations 2o QA/QC 2q References 3) Does the Plan include examples of field labels Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 8 of 17 7) Sample Custody IA iu Nl NA 1) Sample collection- Does the Plan address: 1a) Field custody procedures Transfer of custody and shipment Receipt of samples Sample handlinq/preservation until shipping 1 b Does Plan include examples of custody tracking forms and shipping labels 1c Does the plan address specific mode of shipping (express mail, registered etc.) 2 Does the lab custody procedures: 2a Identify sample custodian 2b Provide for custody record within lab 2c Explain sample receipt procedures and corrective action measures 2d Specify procedures for sample handling, storage, dispersement for analyses, and disposal. 3) Does the Plan address final evidence files Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 9 of 17 8) Calibration Procedures and Frequency Note: Information in this section may be included in detail in the field and analytical methods section but will be checked here to ascertain that it is included in the QAPjP. A table of calibration techniques, appropriate acceptance criteria, and corrective action should be included in this section as reference. IA IU Nl NA 1) For the field and laboratory: 1a Does the Plan include methods/procedures to assure field equipment are functioninq optimally 1b Is schedule/frequency of above included 1c Are equipment logbooks required to record usage, maintenance, calibration, and repair 1d Does Plan include calibration standards or reagents to be used, their source and traceability procedures 1e Does Plan include documentation requirements for calibration: Date (s) of calibration Identification of standards used Results of calibration (raw data, control charts, summary statistics) Corrective action taken Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 10 of 17 9) Analytical Procedures IA IU Nl NA 1) Are all analytical procedure documented or written as SOPs and included in full or by reference for all parameters and include: 1a Scope and Application 1b Method summary 1c Sample handlinq and preservation 1d Interferences 1e Safety 1f Equipment/Materials/Reagents 1g Calibration 1h Procedure 1i Calculations 1j QA/QC 1K References 2) Are the criteria of method selection included (comparability to other project etc.) 3) Are the analytical procedure been approved or equivalent to EPA procedures Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 11 of 17 10) Data Reduction, Validation, and Reporting IA IU Nl NA Reduction 1) Are units specified for all determinations 2) Is raw data saved (where, how) 3) Are equations/procedures used to calculate concentrations included 4) Is the data flow reported 5) QC for data handling reported (backups/frequency) Verification/Validation 1) How will data be verified as acceptable in a consistent manner 2) What methods are used to treat outliers or unacceptable data 3) Data flaqqinq techniques and definitions Reportinq 1) Is the flow or reporting scheme from collection of raw data through document storage included. 2) Are the key individuals who will handle or report data identified. 3) Does the plan describe exactly what is reported Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 12 of 17 11) Internal QC Checks IA iu Nl NA 1) Does the Plan describe procedures for both field and lab 2) Are the protocols used (spikes, blanks splits, etc) described for each parameter and matrix 3) Are field and lab acceptance or control limits specified for each 4) Is the frequency of the checks described 5) Are corrective actions identified 6) Are internal QC check samples/limits/corrective action reported in table format Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 13 of 17 12) Performance and System Audits ia iu Nl NA 1) Are audits addressed: 1a For field activities 2a For lab activities 2) Does the Plan address who will conduct field and lab audits 3) Does the plan describe what protocol will be used for audits 4) Are audit forms developed 5) Are acceptance criteria defined for lab and field activities 6) Does the Plan describe distribution of audit reports 7) Is a schedule of audits included Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 14 of 17 13) Preventative Maintenance IA IU Nl NA 1) Does the Plan include a maintenance schedule to minimize down time 1a For field activities 1b For Lab activities 2) Is a spare parts list available 3) Is a source of spare parts identified 4) Is the source of repair described Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 15 of 17 14) Specific Routine Procedures to be Used to Assess Data Precision, Accuracy, and completeness of Specific Measurement Parameters Involved IA IU Nl NA 1) Relative to the objectives in Section 5, does the Plan include protocols for monitorinq whether requirements were met 2) Does the Plan include equations used to calculate precision, accuracy, and completeness 3) Does the Plan describe the methods used to gather information for precision, accuracy calculations 4) Are the statistical procedures used documented Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 16 of 17 15) Corrective Action IA IU Nl NA 1) Does the plan include a scheme to: 1a Identify major sources of error 1b Plan and implement correction 1c Document results of process 2) Does the plan include predetermined limits for data acceptability beyond which corrective action is required 3) Are procedures for corrective action (who initiates, who approves) included 4) Is feedback from systems audits (lab and field ) addressed Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments: ------- Appendix C Revision 0 Date: 5/96 Page 17 of 17 16) Quality Assurance Reports to Management IA iu Nl NA 1) Does the Plan specify the type (progress, final) and frequency of reporting 2) Does the Plan address: 2a Status of project (time table) 2b Results of performance and systems audits 2c Data quality assessment 2d Siqnificant QA problems and proposed corrective action 2e Chanqes in QAPjP (methods etc.) 3) Final report and distribution 3a) Contents of final report Note IA = Included & Acceptable IU = Included & Unacceptable or Questionable Nl = Not Included NA = Not Applicable/Necessary Comments ------- |