INTERIM DRAFT EPA REQUIREMENTS FOR QUALITY ASSURANCE PROJECT PLANS EPA QA/R-5 Region 6 U.S. EPA Office of Quality Assurance Dallas, Texas 75202 MAY 1994 ------- QUALITY ASSURANCE PROJECT PLANS Introduction This document incorporates the national U.S. EPA Requirements for Quality Assurance Project Plans (QAPP) and the Region 6 policy requirements into one document for ease of use by our customers. It is both a Regulatory requirement and policy of EPA that all projects and tasks involving environmentally related measurements shall have a QAPP developed and approved by the Agency prior to any such activities being accomplished. This chapter presents detailed instructions on what information must be contained in a Quality Assurance Project Plan (QAPP) for environmental data operations performed by or on behalf of EPA Region 6 and on the procedures for its review and approval. While there is diversity in the various programs, it is Region 6 policy that all elements applicable to the QAPP Use Category, to be discussed in detail further on in this document, will be addressed in each QAPP. To do otherwise would be to deviate from the national standard and could dilute EPA's goal of national consistency for Quality Assurance requirements. The complexity of environmental data operations demands that a systematic process and structure be established to provide decision makers with the necessary confidence in the quality of data produced for the decisions to be made, as well as with the means to determine when the data are not fully usable and what to do about the situation. This process and structure is provided by the quality system for the organization conducting the environmental data operations. EPA policy requires that the collection of environmental data by and on behalf of the Agency be supported by a mandatory Quality System, documented in a Quality Management Plan (QMP). An area that must be addressed in each QMP is the systematic planning process used for each project or task such as the Data Quality Objective process. All projects involving the generation, acquisition and use of environmental data shall be planned using a systematic planning process such as the Data Quality Objective process as defined by the current revision of Guidance for Planning for Data Collection in Support of Environmental Decision Making Using the Data Quality Objectives Process, EPA QA/G-4, or acceptable alternate. The use of the Data Quality Objectives process leads to definition of many of the specific elements required in a QAPP. It is EPA policy that all decisions and work involving the use of environmental data be supported by a Quality Assurance Project Plan. The QAPP is the principal product of the planning process inasmuch as it integrates all technical and quality aspects for the life-cycle 1 ------- of the project, including planning, implementation, and assessment. The QAPP provides a project or task-specific blueprint of how QA and QC are applied to an environmental data operation to assure that the results obtained are of the type and quality needed and expected. Effective implementation of the QAPP should provide the best opportunity to achieve the technical and quality goals of a project. The ultimate success of an environmental program or project depends on the quality of the environmental data collected and used in decision-making, and this may depend significantly on the adequacy of the QAPP and its effective implementation. Quality planning is an absolutely essential component of project management and the QAPP provides the mechanism for documenting the results of the planning process. In the sections to follow, the elements of the QAPP are discussed in detail. These elements represent the information that EPA believes to be necessary for data operations involving the character- ization of environmental processes and conditions. Quality Assurance Project Plan Requirements QAPP Policy All work performed by or on behalf of EPA that involves the collection and use of environmental data1 shall be supported by an Agency-approved Quality Assurance Project Plan (QAPP). No work performed under contract, work assignment, technical directive, delivery order, assistance agreement, or inter-agency agreement involving the environmental data generated from direct measurement activities, collected from other sources, or compiled from computerized data bases and information systems shall be implemented without an ap- proved QAPP or without a condition stating that the actual collection of data shall not occur until after the QAPP is approved. Region 6 has implemented this requirement by establishing a process that requires a joint certification of the presence and approval status of QMPs and QAPPs by the Region 6 Office of Quality Assurance and the Region 6 Program Office that has management responsibilities for the particular project. QAPP Purpose The QAPP is a critical planning document for any environmental data operation. The QAPP documents how QA and QC activities will be planned, implemented, and assessed during the life cycle of a program, project, or task. The QAPP is the "road map" or blueprint for how a particular project is integrated with the QA program of the organization performing the work and how the specific QA and QC activities will be applied during a particular project. Environmental data include any information collected, produced, or derived from measurements, analyses, or models of environmental processes and conditions, or from experimental systems representing such processes and conditions, including results from laboratory analyses. ------- Applicability of QAPPs These QAPP requirements apply to all (intramural and extramural) environmental data operations that acquire, generate, or compile environmentally-related data and that are per- formed by or on behalf of EPA. These operations include work performed through con- tracts, interagency agreements, and financial assistance agreements (e.g., cooperative agree- ments, grants). Environmental data operations encompass diverse and complex activities, and represent efforts pertaining to rulemaking, compliance with regulations, and research. Consequently, any requirements developed to define how QA/QC should be applied to environmental activities must contain considerable flexibility. This may mean, for example, that some environmental data operations, perhaps involving research projects, may only require a qualitative discussion of the experimental process and its objectives, such as a project narrative statement. Others may require extensive documentation in order to adequately describe a complex environmental program. This means that the content and level of detail in each QAPP will vary according to the nature of the work being performed and the intended use of the data. For QAPPs submitted to EPA Region 6 the decision on the content and level of detail is to be consistent with national EPA requirements, as defined by the current revision of EPA Requirements for Quality Assurance Project Plans for Environmental Data Operations, EPA QA/R-5. It is EPA policy that the type and quality of environmental data needed for their intended use shall be defined and documented using the EPA Data Quality Objectives (DQO) Process, or its equivalent, to the extent possible or practicable. The results of the DQO Process provide key inputs to the QAPP and will largely determine the level of detail required in the QAPP. These inputs from the DQO Process are identified later. QAPP Use Categories Because of the diversity and variability in the mission requirements of the organiza- tions (e.g., state program offices, EPA regions, research laboratories, municipal organizations) comprising the environmental community, it is not always possible to define a single checklist of elements and details needed for all QAPPs. To provide some of the needed flexibility, several EPA organizations, including Region 6, have been categorizing QAPP requirements according to the type of work being performed and the intended use of the data. Four categories have been defined that vary the level of detail and rigor prescribed for particular QAPPs. These Use Categories are mandatory for QAPPs prepared and submitted for approval to U.S. EPA Region 6. They are presented as an aid to determining the level of detail that may be needed in a QAPP for a particular type of work. This approach recognizes that not all environmental data operations require QAPPs with the same level of detail. For example, data collected for compliance or enforcement decisions in a Region will ------- require a more comprehensive QAPP than an exploratory research project conducted for an ORE) R&D laboratory. The categories are: • Category I - Direct Support to Rulemaking, Enforcement, Regulatory, or Policy Decisions These projects include environmental data operations that directly support rulemaking, enforcement, regulatory, or policy decisions. They also include research projects of significant national interest, such as those typically monitored by the Administrator. Category I projects require the most detailed and rigorous QA and QC for legal and scientific defensibility. Category I projects are typically stand-alone; that is, the results from such projects are sufficient to make the needed decision without input from other projects. • Category n - Complementary Support to Rulemaking, Regulatory, or Policy Decisions These projects include environmental data operations that complement other projects in support of rulemaking, regulatory, or policy decisions. Such projects are of sufficient scope and substance that their results could be combined with those from other projects of similar scope to provide the necessary information for decisions. Category II projects may also include certain high-visibility projects as defined by EPA management. • Category HI - Interim Studies These projects include environmental data operations performed as interim steps in a larger group of operations. Such projects include testing research hypotheses, estimating effects, developing methods, and other work producing results that are used to evaluate and select options for interim decisions or to perform feasibility studies or preliminary assessments of unexplored areas for possible future work. • Category IV - Basic Studies These are projects involving environmental data operations to study basic phenomena or issues, including proof of concept and qualitative screening for particular analytical species. The final determination of a project's category is made by the EPA Region 6 ------- Program Office2 responsible for the work. Where no category is defined, the preparer of the QAPP shall assume that it is a Category I project. It should be noted that projects may contain specific tasks or subtasks that vary in the level of QA/QC requirements. Such conditions should be considered when deciding on the Use Category for a particular project. Special Requirements In some cases, it may be necessary for special requirements to be added to the QAPP that are not included in this document. When this occurs, such additional elements must also be addressed by the QAPP in the same manner as the elements listed in this document. The EPA Region 6 program office sponsoring the work shall delineate any specific requirements beyond those listed in this manual. If none are specified, the QAPP shall address all elements required, as specified by their particular Use Category. Attached documentation, such as an approved Work Plan, Standard Operating Procedures (SOPs), etc., may be referenced in response to a particular QAPP element. This is, in fact, encouraged to reduce the time required to prepare the QAPP. The QAPP shall also address related QA planning documentation (e.g., Quality Management Plans, QA Project Plans) from subcontractors or suppliers of services critical to the technical and quality objectives of the project or task. In any case, all referenced documents must be attached to the QAPP itself or be placed on file with the appropriate EPA office for routine referencing when needed. Such references must be kept current by the submitter. QAPP Process Responsibilities and Approvals The QAPP may be prepared by different groups. The QAPP may be prepared by the EPA principal investigator for an in-house project and may be reviewed by the principal investigator's immediate supervisor before it goes to the QA manager or coordinator for final approval. The QAPP may be prepared by a contractor or an assistance agreement holder. In addition, the QAPP may be prepared by another Federal agency under an interagency agreement. These specific situations must be addressed in the organization's QMP to establish how, when, and by whom review, approval, and effective oversight of QAPPs occurs. Except where specifically delegated, all QAPPs prepared by non-EPA organiza- tions must be approved by EPA for implementation. Region 6 policy allows for the development of a QAPP to occur only after a QMP has been approved, and specifically precludes approval of a QAPP until the applicable QMP has been approved. EPA believes that the necessary flexibility in content and level of detail in the QAPP may be best achieved by having the QAPP requirements reviewed and confirmed by the EPA Organization refers to the EPA Program Office, Region, or ORD Laboratory having an approved Quality Management Plan that describes its quality system for planning, implementing, and assessing environmental programs. ------- project manager (or officer)3 with the assistance and concurrence of the EPA QA Manager. In the case of Region 6, the authority to review and approve QAPPs has been delegated to the Region 6 Program Offices, and is covered and defined by the Quality Management Plans of the various divisions. None of the environmental work addressed by the QAPP should be started until the initial QAPP has been approved and distributed to project personnel. It is Region 6 policy that sixty days prior to the initiation of any environmental measurements or data generation, the recipient (of any EPA funding) shall submit to the EPA Project Officer, for review and approval, a written Quality Assurance Project Plan for the project. Any costs for environmental measurements or data generation incurred prior to approval of the QAPP by the EPA Project Officer will be ineligible for reimbursement. The Region 6 Policy does not authorize granting conditional approval to a QAPP. There are only two recognized statuses of a QAPP are approved, and not approved. It is the responsibility of the organization performing the work to assure that no environmental data are acquired before the QAPP is approved and received by project personnel. QAPP Implementation and Revision It is EPA policy that all approved QAPPs shall be implemented for the intended work. It is the responsibility of the group performing the work to implement the approved QAPP and to ensure that all personnel involved in the work have copies of the approved QAPP and all other necessary planning documents and understand their requirements prior to the start of data generation activities. Because of the complex and diverse nature of environmental data operations, changes to original plans are often needed. When such changes occur, it is the responsibility of the Project Manager for whom the work is being performed to determine if the change signifi- cantly impacts the technical and quality objectives of the project. This determination should be done in consultation with the QA Manager. When substantive change is contemplated, the originator of the QAPP shall modify the QAPP as necessary to document the change and submit the revision for approval by the same parties as for the original review. Only after the revision has been approved and receipt of the change (at least verbally with written follow-up) by the project personnel performing the work, shall the change be implemented. It is absolutely essential that the QAPP be kept current and that all personnel involved in the work effort have a current version of the QAPP available. For programs or projects of long duration, such as multi-year monitoring programs, the QAPPs shall be reviewed at least annually by the Project Manager, revised if necessary to reflect current needs, and 3This term refers to the responsible EPA official for the project and includes such descriptors as project officer, work assignment manager, and principal investigator. ------- resubmitted for review and approval. If the entire QAPP is current, valid and accurately reflects the project goals and the organization's policy, at a minimum, each year the organization will submit to EPA Region 6 a certification that the plan is current, to include a copy of new, signed approval pages for the QAPP. OAPP Elements General Content All projects involving the generation, acquisition, and use of environmental data shall be planned and documented. The QAPP is the logical product of the planning process and must provide sufficient detail to demonstrate that: • the project technical and quality objectives (e.g., DQOs, or equivalent) are identified and agreed upon; • the intended measurements or data acquisition methods are appropriate for achieving project objectives; • assessment procedures (including QA and QC) are sufficient for obtaining data of the type and quality needed and expected; and • any limitations on the use of the data can be identified and documented. Environmental data operations require the coordinated efforts of many individuals, including managers, engineers, scientists, chemists, statisticians, and others. The QAPP must integrate the contributions and requirements of everyone involved into a clear, concise "blueprint" of what is to be accomplished, how it will be done, and by whom. This means it must provide understandable instructions to those who must implement all parts of the QAPP, including the field sampling team, the analytical laboratory, and the data reviewers. In all aspects of the QAPP, the use of national standards and practices, and inclusion of standard operating procedures is encouraged. The QAPP elements that follow are presented in an order corresponding to planning, implementation, and assessment. They have been grouped for convenience into four types of elements. The individual QAPP elements are numbered sequentially within each group. The four types of elements and their intent are summarized as follows: A Project Management - This group of QAPP elements covers the basic area of project management, including the project history and objectives, roles and responsibilities of the participants, etc. These elements ensure that the project has a goal, that the participants understand the goal and the approach to be used, and that project planning is documented. ------- B Measurement/Data Acquisition - This group of QAPP elements covers all of aspects of measurement systems design and implementation, ensuring that appropriate methods for sampling , analysis, data handling, and QC are employed and will be thoroughly documented. C Assessment/Oversight - This group of QAPP elements addresses the activities for assessing the effectiveness of the implementation of the project and associ- ated QA/QC. The purpose of assessment is to ensure that the QAPP is implemented as prescribed. D Data Validation and Usability - This group of QAPP elements covers the QA activities that occur after the data collection phase of the project is completed. They ensure that the individual data elements conform to the specified criteria, thus enabling reconciliation with the project objectives. The Use Categories and elements are summarized in Table 4-1. As stated earlier, the decision to use this approach and the determination of a project's category and the actual QAPP elements required is made by the EPA organization. Region 6 requires the Use Category Codes be utilized in QAPPs. The term DQO after the QAPP Element title indicates that the information needed to address this element in the QAPP is usually identified in the DQO process. TABLE 4-1 USE CATEGORY CHART CATEGORY ELEMENT DESCRIPTION PROJECT MANAGEMENT I, II, III, IV Al I, II, III A2 I, II, III, IV A3 I, II, III, IV A4 I, II, III, IV A5 I, II, III A6 I, II, III A7 IV A8 I A9 I, II, III A10 Title and Approval Sheet Table of Contents Distribution List (DQO) Project/Task Organization (DQO) Problem Definition/Background (DQO) Project/Task Description (DQO) Data Quality Objectives for Measurement Data Project Narrative (ORD Only) Special Training Requirements/Certification Documentation and Records ------- TABLE 4-1 USE CATEGORY CHART (CONTINUED) CATEGORY ELEMENT DESCRIPTION MEASUREMENT/DATA ACQUISITION I, II, HI I, II, III I, II, III I, II, III I, II, III I, II I, II, III I I, II, III I, II Bl (DQO) Sampling Process Design (Experimental Design) B2 Sampling Methods Requirements B3 Sample Handling and Custody Requirements B4 (DQO) Analytical Methods Requirements B5 Quality Control Requirements B6 Instrument/Equipment Testing, Inspection, and Maintenance Requirements B7 Instrument Calibration and Frequency B8 Inspection/Acceptance Requirements for Supplies and Consumables B9 Data Acquisition Requirements (Non-direct Mea- surements) BIO Data Management ASSESSMENT/OVERSIGHT I, II, III I, II, III Cl C2 Assessments and Response Actions Reports to Management DATA VALIDATION AND USABILITY I, II, III I, II I, II, III Dl D2 D3 Data Review, Validation, and Verification Re- quirements Validation and Verification Methods Reconciliation with Data Quality Objectives ------- Group A, Project Management Al Title and Approval Sheet Include: Title of the plan Name of the organization(s) implementing the project Names, titles, signatures of appropriate approving officials and approval dates for: Organization's Project Manager (Required) Organization's Quality Assurance Manager (Required) Project Subordinate Supervisors Concurrence (Optional) Region 6 EPA Project Manager (Required) Region 6 EPA Approving Official (Required)4 Others, as needed (e.g., State, other Federal Agency) Title and Region 6 QTRAK number of the approved Quality Management Plan applicable to submitted QAPP Submission of at least two original approval pages is recommended. A2 Table of Contents List the sections, figures, tables, references, and appendices. Document control format may be required at the option of the Project Manager and QA Manager, and is encouraged by EPA Region 6. When required, use document control format in the upper right-hand corner of each page following the Title and Approval Sheet. For example: Section No. Revision No. Date Page of 4 "EPA Approving Official" is the Region 6 Program Office Manager or staff person designated and authorized by Certification of the Region 6 QA Officer to approve QAPPs. If the EPA Project Manager/Officer is unable to determine the approving official, contact the Office of Quality Assurance. 10 ------- A3 Distribution List List the individuals and their organizations who will receive copies of the approved QAPP and any subsequent revisions. Include all managers who are responsible for implementing the plan, as well as the QA managers and representatives of all groups involved. A4 Project/Task Organization (DQO) Identify the individuals or organizations participating in the project and discuss their specific roles and responsibilities. Include the principal data users and the decision-makers. The project quality assurance manager shall be indepen- dent of the unit generating the data. This does not include senior officials, such as corporate managers or agency administrators, who are nominally but not functionally involved in data generation, data use, or decision-making. The QAPP should also identify the person(s) responsible for approving and accepting final products and deliverables. Provide a concise organization chart showing the relationships and the lines of communication among all project participants. Include other data users who are outside of the organization generating the data, but for whom the data are nevertheless intended; e.g., modelers, risk assessors, design engineers, toxicologists, etc. Where direct contact between project managers and data users does not occur, such as, between a project consultant for a Potentially - Responsible Party and EPA risk assessment staff, the organization chart should show the route by which information is exchanged. The organization chart should also identify any subcontractor relationships relevant to environmental data operations. This chart should be realistic and practical, and should reflect only actual lines of authority and communication for the project described. Names of current incumbent occupying a position is essential, as is the identification of vacant positions. A5 Problem Definition/Background (DQO) State the specific problem to be solved or decision to be made. Include sufficient background information to provide a historical perspective for this particular project. For example, this would include the regulatory or alleged toxic exposure situation that led to the need for this project. The discussion must include enough information about the problem, the past history, any previous work or data, and any regulatory or legal context to allow a techni- cally-trained reader to make sense of the project objectives and activities. This discussion also identifies the decision maker(s) and the principal customer(s) 11 ------- for the results. (When the Data Quality Objectives [DQO] process has been used, this information should be readily available.) A6 Project/Task Description (DQO) Provide a description of the work to be performed. This discussion may not need to be lengthy or overly detailed, but it should give a overall picture of how the project will resolve the problem or question described in A5. Describe in general terms the following, as needed: • Measurements that are expected during the course of the project and the approach that will be used. • Applicable technical, regulatory, or program-specific quality standards, criteria, or objectives. • Any special personnel and equipment requirements that may indicate the complexity of the project. • The assessment tools needed (i.e., program technical reviews, peer reviews, surveillances, and technical audits as needed and/or specified by the QMP) for the project. • A schedule for the work to be performed. • Project and quality records required, including the types of reports needed. A7 Quality Objectives and Criteria for Measurement Data (DQO) Any QAPP must include a statement of the project quality objectives and measurement performance criteria. EPA recommends that a graded approach be used in planning, such as the Data Quality Objectives (DQO) Process. Even in those cases in which the formal DQO Process is not needed, a statement of the project quality objectives and measurement performance criteria is needed. The DQO process provides quality objectives based on several factors chosen by the user of the data. For details on the DQO Process and when it should be used, please see the EPA guidance document (QA/G-4)5. 5 EPA QA/G-4 is Guidance for Planning for Data Collection in Support of Environmental Decision Making Using flie Data Quality Objectives Process. 12 ------- The project quality objectives should be stated in quantitative terms to the extent possible: Example: UV Treatment of Contaminated Groundwater. "The purpose of this project is to demonstrate whether or not the residual trichlorethene concentration in the treated water is less than 0.5 /xg/L at a confidence level of 95 percent." Without such quantitative goals, it is difficult to know whether the selected analytical method is sufficiently sensitive or precise, or whether a sufficient number of samples are being collected. Sometimes, of course, project objec- tives must be stated somewhat less quantitatively, particularly in those situa- tions when the use of the formal DQO Process is not needed. Example: Oil Spill Remediation. "One objective of this project is to determine whether the populations of clams, mussels, and sand fleas recover more rapidly in the treated than the untreated area." Example: Epidemiology Study. "The purpose of this project is to determine whether the concentrations of indoor NO2 and soot are greater in the residences of lung cancer patients than in the residences of healthy persons." The section on project quality objectives and measurement performance criteria should address, as appropriate, the following: • the scope of the project; that is, the domain (geographical locale and boundaries, environmental medium, time period, etc.) over which conclusions and decisions will apply; • the time, resource, or other constraints on the measurement project; • the intended uses of the data, in order of importance and the expected users of the data; • the specific data needed: type, quantity, matrices involved; • the action levels or standards upon which decisions will be made, including the detection limits and data reporting units, and the source(s) of this information; • the population parameter(s) of interest; e.g., mean, maximum, or range, which specify the form the data will be in when compared against action levels or standards; 13 ------- • the acceptable level of confidence in the data needed for the stated purposes or the acceptable amount of uncertainty; • the quantitative sensitivity, precision, bias, and completeness criteria for each major measurement planned (including all pollutant and process measurements) for each sample matrix, based on the DQO statements; • the units of expression of the precision and bias goals, which should correspond to the methods selected to assess data precision and bias; and • the goals for achieving data representativeness and comparability, and the planning considerations for attaining these goals (unlike precision, bias, and completeness, these objectives are not usually expressed or assessed quantitatively); Data quality or measurement performance criteria may be typically specified in terms of detection or quantitation limits, precision, bias, and comparability. However, simply listing requirements for precision, bias, and completeness without further discussion is not sufficient. Even statements such as "bias will be measured as percent recovery of a matrix spike sample" are of marginal help. In specifying data quality, it is thus essential to specify exactly how such quality will be measured and interpreted. Example: A possible statement of bias requirement. "Bias will be mea- sured a minimum of five times throughout this project by the analysis of standard reference materials No. 956B. Recovery of TCE from this SRM should average 85 percent or greater, with a relative standard deviation of no more than 20 percent." Data specifications should also be distinguished from the specific QC proce- dures that are routinely carried out as part of each measurement. QC proce- dures are used while carrying out specific procedures; data specifications are used for selecting the appropriate methods and QC criteria. The QAPP may need to define different types of sensitivity (i.e., qualitative, quantitative, screening, etc.) that may be appropriate for different parts of the project. The quantitative goals should reflect the total measurement, if possible, or address the field, laboratory, and data handling components separately. In the event there is no basis for defining data quality goals for the project, goals 14 ------- may be estimated based on prior knowledge of the measurement system, and on method validation studies (using replicates, spikes, standards, recovery, studies, etc.) Explain the circumstances under which these goals were estab- lished. If defining quantitative goals is not relevant for certain measurements, indicate this and state the reason. Data representativeness is the degree to which the environmental samples truly reflect the population or material in the real world. It can be affected by the time, place, and manner by which the samples are collected. Data comparability is dependent upon consistency in sampling conditions, selection of sampling procedures, sample preservation methods, analytical methods, and data reporting units, throughout the project, and with previous projects with which these results will be compared. The DQO Process for compliance and/or enforcement projects in the Regions may not be within the control of the EPA project manager and QA manager. The measurement performance criteria are specified in regulations, permits, or orders. Whether or not those criteria satisfy the requirements of this docu- ment, they are absolutely required. Often the results of the DQO Process are not expressible in the terms stated here, such as precision, bias, or compara- bility. The affected source is simply required to follow a specified method. In such cases, it is sufficient for the portion of the QAPP addressing the DQO Process to state that the testing will satisfy the regulatory requirements speci- fied. This does not, however, relieve the project manager or QA manager from their responsibility to comply with all other applicable QAPP require- ments in this document. A8 Project Narrative Discuss in a narrative form the following issues as they pertain to the project or task, as needed: work to be performed or hypothesis to be tested, anticipated use of the data, how (quantitatively or qualitatively) the success of the project or task will be determined (A7, D3), survey design requirements and description (Bl), sample type and sampling location requirements (B2), sample handling and custody requirements (B3), 15 ------- selection of analytical methods (B4), calibration and performance evaluation samples for sampling and analytical methods used (B5), sampling or analytical instrumentation requirements (B6), plans for peer or readiness reviews prior to data collection (Cl), and any on-going assessments during actual operation (oversight) (Cl). QAPP elements corresponding to the items to be addressed in the narrative are given in parentheses. The narrative should allow technical or QA readers to relate the project or task to the DQOs and to the Problem Definition stated earlier in the QAPP. (The use of the Project Narrative is consistent with the Use Category scheme described in Appendix A. Since this element addresses in narrative form many other QAPP elements, it is not necessary to specifical- ly include those elements in a Category IV QAPP. Moreover, because the elements listed above would be addressed in more detail in Category I, II, and III QAPPs, it is not necessary to include this element for those categories.) A9 Special Training Requirements/Certification Identify and describe any specialized training or certification requirements for personnel in order to successfully complete the project or task. Discuss how such training will be provided and how the necessary skills will be assured and documented. A10 Documentation and Records Itemize the information and records which must be included in a data report package for the project or task, and specify the reporting format, if desired. Documentation can include raw data, filed logs, instrument printouts, and results of calibration and QC checks. Specify the laboratory data reporting turnaround time. Specify whether a field sampling and/or laboratory analysis "case narrative"6 is required to provide a complete description of any difficul- ties encountered during sampling or analysis. Specify any requirements for the final disposition of records and documents from the project, including location and length of retention period. 6"Case Narrative" refers to an annotated summary of the analytical work performed by a laboratory that describes in narrative form what activities were performed and identifies any problems encountered. The case narrative provides additional information to user in interpreting the data received. 16 ------- Group B, Measurement/Data Acquisition The following QAPP elements describe the requirements related to the actual methods to be used for the: • collection, handling, and analysis of samples; • direct measurement of parameters that do not require the actual collec- tion of a sample; and • the management (i.e., compiling, handling) of the data. The methods described in these elements may have been cited earlier in element A6; however, the purpose here is to include sufficient detailed information to ensure that the methods are verified and documented. As long as the designated methods are well docu- mented and are readily available to all project participants, specific citations are adequate. If these methods are not readily available, detailed copies of the methods and/or SOPs must accompany the QAPP as attachments. Bl Sampling Process Design (Experimental Design) (DQO) Outline in general terms the experimental design of the project and the antici- pated project activities, including the types of samples required, sampling network design, sampling frequencies, sample matrices, measurement parame- ters of interest, and the rationale for the design. If individual sampling plans will be developed for discrete project phases, include their preparation sched- ule. Describe techniques or guidelines to be followed in selecting sampling points and frequencies, well installation design (when applicable), field decontamina- tion procedures and materials needed, and sampling equipment. When field screening techniques will be used to identify samples for laboratory analysis, describe the criteria for sample selection. Similarly, when locational data are to be collected, stored, and transmitted, the method(s) used must be specified and described (or referenced). Key elements to be addressed include how locations and their bias are determined. All measurements should be classified as critical (i.e., required to achieve project objectives) or non-critical (informational purposes only). Critical measurements will undergo closer scrutiny during the review and data gather- ing process, and will have first-claim on limited budget resources. For non-standard methods or unusual sample matrices and situations, appropri- ate method validation study information is needed to confirm the performance 17 ------- of the method for the particular matrix. Such validation studies may included round-robin studies performed by other organizations. If previous validation studies are not available, they must be developed during the project and included as part of the project results. It is very important for this element to include complete documentation and validation of both the sampling and analytical methodologies. Identifying standard methods by number, date, and regulatory citation (as appropriate) is often sufficient. However, many published (and even regulatory) methods allow the user to select from various options. The method citations should state exactly which options are being selected. Measurement of process conditions is often essential to a project (e.g., indus- trial plant or control equipment operation associated with a compliance test, meteorological parameters associated with impoundment volatization). In such cases, the experimental design must include the design and validation tech- niques as described above. B2 Sampling Methods Requirements Describe the procedures for collecting samples. Identify the required sampling methods (and/or equipment, if automated), including any implementation requirements, decontamination procedures and materials needed, and any specific performance requirements for the method. For each sampling method, identify any support facilities needed. The discussion should also address what to do if there is a failure in the sampling or measurement system and who is responsible for corrective action. Describe the preparation and decontamination of sampling equipment, includ- ing disposal of decontamination by-products; the selection and preparation of sample containers, sample volumes, preservation methods, and maximum holding times to sample extraction and/or analysis. A tabular presentation format is strongly recommended, particularly when two or more sample matrices are involved. B3 Sample Handling and Custody Requirements Describe the provisions for sample handling and shipment, taking into account the nature of the samples and the maximum allowable sample holding times before extraction or analysis. Describes the following provisions for sample custody, in both the field and the laboratory: 18 ------- Forms, notebooks and procedures to record the exact location and ambient conditions associated with sample collection, possession and analysis. In the laboratory, a sample custody log should be maintained. Examples of sample documentation forms, such as sample labels, custody seals, and chain-of-custody forms. Labeling procedures and information entered on the forms, including sample preservation, if any, and dates and times of sample transfer and analysis. Procedures for transferring and maintaining custody of samples. B4 Analytical Methods Requirements (DQO) Identify the analytical methods and/or equipment required, including any extraction methods needed, laboratory decontamination procedures and materi- als needed (such as in the case of hazardous or radioactive samples), waste disposal requirements (if any), and any specific performance requirements for the method. The QAPP should also address what to do if there is a failure in the analytical system and who is responsible for corrective action. B5 Quality Control Requirements Discuss QC procedures that should be associated with each sampling, analysis, or measurement technique. Such specific procedures are performed routinely during the measurement process, and the results are required to be evaluated immediately by the technician upon completion of the test. Results must fall within certain acceptance criteria, or specific corrective action is required. For projects at or beyond the "proof-of-concept" stage, or for projects employ- ing well-characterized methods, this section should list each required QC procedure, along with the associated acceptance criteria and corrective action. Because standard methods are often vague or incomplete in specifying QC requirements, simply relying on the cited method to provide this information is usually insufficient. In any case, QC procedures must frequently be modified on a project-specific basis in order to meet data specifications. QC procedures must be compatible with the data specifications discussed above. This means, if a measurement must be precise within ±20 percent, the stability of calibration checks must be somewhat better than +20 percent. For some research-oriented projects, the analytical technique may not be available until well into the project. In such instances, detailed QC requirements may 19 ------- not need to be specified in the initial QAPP. More appropriately, the initial document might specify general requirements for precision, bias, and detection limits, and the means of achieving these goals would be developed by the principal investigator during the course of the project. List the required QC checks, such as matrix spikes, duplicates, blanks, laboratory control samples, surrogates, or second column confirmation. State the frequency of analysis for each type of QC check, and the spike compounds and levels. State or reference the required control limits for each QC check and corrective action required when control limits are exceeded. Describe the procedures to be used to calculate each of the QC statistics, including the QC checks described in the preceding paragraph as well as precision and bias. Copies of the formulas are acceptable as long as the accompanying narrative or explanation specifies clearly how the calculations will address difficult situations such as missing data values and "less than" or "greater than" values. B6 Instrument/Equipment Testing, Inspection, and Maintenance Re- quirements Discuss how inspections and acceptance testing, including the use of QC standards and reference materials, of environmental sampling and measurement systems and their components must be performed and documented to assure their intended use as specified by the design. Identify and discuss how final acceptance shall be performed by independent personnel (e.g., personnel other than those performing the work). Discuss how deficiencies are to be resolved when acceptance criteria are not met, and how and when re-inspection will be performed as necessary. Discuss how periodic preventive and corrective maintenance of measurement or test equipment shall be performed to ensure availability and satisfactory performance of the systems. Identify the equipment and/or systems requiring periodic maintenance. Discuss how the availability of critical spare parts, identified in the operating guidance and/or design specifications of the systems, will be assured and maintained. B7 Instrument Calibration and Frequency Identify all tools, gauges, instruments, and other sampling, measuring, and test equipment used for data collection activities affecting quality that must be controlled and, at specified periods, calibrated to maintain bias within specified 20 ------- limits. Discuss how calibration shall be conducted using certified equipment and/or standards with known valid relationships to nationally recognized performance standards. If no such nationally recognized standards exist, document the basis for the calibration. Identify the certified equipment and/or standards used for calibration. Indicate how documentation of calibration shall be maintained and be traceable to the instrument. B8 Inspection/Acceptance Requirements for Supplies and Consumables Discuss how and by whom supplies and consumables shall be inspected and accepted for use in the project. Supplies and consumables are those items necessary to support the sampling and analytical operation. They include, but are not limited to: sample bottles, calibration gases, reagents, hoses, materials for decontamination of sampling equipment, deionized water, and potable water. Identify acceptance criteria for such supplies and consumables in order to satisfy the technical and quality objectives of the project or task. B9 Data Acquisition Requirements (Non-direct Measurements) Identify the type of data acquired from non-measurement sources such as computer data bases, spreadsheets, and programs, and literature files. Define acceptance criteria for the use of the data in this project. Discuss any limita- tions on the use of the data based on uncertainty in the quality of the data and discuss the nature of that uncertainty. BIO Data Management Outline the project data management scheme, tracing the path of the data, beginning from receipt from the field or laboratory, to the use or storage of the final reported form. Describe the standard record-keeping procedures, document control system, and the approach used for data storage and retrieval on electronic media. Discuss the control mechanism for detecting and correct- ing paperwork errors and for preventing loss of data during data reduction (i.e., calculations), data reporting, and data entry to forms, reports, and databases. Provide examples of any forms or checklists to be used. Identify and describe all data handling equipment and procedures that will be used to process, compile, and analyze the data. This includes procedures for addressing data generated as part of this project as well as data from other sources. The specifications should include any required computer hardware and software and should address any specific performance requirements for the 21 ------- hardware/software configuration used. Describe the procedures that will be followed to demonstrate acceptability of the hardware/software configuration required. Group C, Assessment/Oversight Cl Assessments and Response Actions Identify the number, frequency, and type of assessment activities needed for this project. Assessments include, but are not limited to, the following: surveillance, peer review, management systems review, readiness review, technical systems audit, performance evaluation, audit of data quality, and data quality assessment. Discuss the information expected from the assessment and success criteria (i.e., goals, performance objectives, acceptance criteria specifications, etc.) for each assessment proposed. For each proposed assessment, list the approximate schedule of activities, and discuss the information expected from the assess- ment and the criteria for success. For any planned self-assessments (utilizing personnel from within the project groups), identify the participants and their exact relationship within the project organization. For independent assess- ments, identify the organization and person(s) that will perform the assess- ments. Discuss how and to whom the results of the assessments will be reported. Define the authorities of the assessors. For example, if the asses- sors should order a work suspension upon finding a significant condition, this section delineates clearly their authority to do so. Define explicitly the unsatisfactory conditions under which the assessors are authorized to act. Recognizing that assessments may be needed at any time during the project, provide a schedule for the assessments to be performed. Discuss how response actions to non-conforming conditions will be addressed and by whom. Identify who is responsible for implementing the response action. Describe how response actions will be verified, validated, and docu- mented. 22 ------- C2 Reports to Management Identify the frequency, content, and distribution of reports issued to inform management of the following: • status of the project; • results of performance evaluations and system audits; • results of periodic data quality assessments; and • significant quality assurance problems and recommended solutions. Identify the responsible organization(s) that will prepare the reports, and the recipients of the reports. Identify any other status reports to management as well as their content and frequency. Group D, Data Validation and Usability Dl Data Review, Validation, and Verification Requirements State the criteria used to review and validate - that is, accept, reject, or qualify - data, in an objective and consistent manner. Provide examples of any forms or checklists to be used. Project-specific calculations or algorithms should be discussed. Some projects may require special calculation during or after data generation: Example: Indoor Air Pollution. Consider a project that is meant to estimate the number of residences within the greater Washington, D. C., area exhibiting N02 concentrations greater than 100 /xg/m3 at a frequency of 30 of more days per year. Once NO2 measurements are available, one would attempt to extrapolate the limited information to the greater metropolitan area. The QAPP should explain the statistical techniques that will be employed, including how uncertainties will be assessed. For other projects, one may only need to calculate a mass balance of ade- struction/removal efficiency. While these are much simpler requirements, the specific formulas and data inputs should be listed. This approach helps assure that at least the minimum necessary data are collected for the intended interpretation (even if additional interpretation schemes are eventually employed). 23 ------- D2 Validation and Verification Methods Describe the process to be used for validating and verifying data, including the chain of custody for data throughout the life cycle of the project or task. Discuss how issues shall be resolved and the authorities for resolving such issues. Describe how the results are conveyed to data users. The review of data can include checks of the following: transmittal errors, field and labora- tory QC data, detection limits, instrument calibration, special sampling or analysis conditions, performance evaluations, technical systems audits, contract compliance issues (e.g., holding times), and statistical data treatments, such as tests for identification of potential outliers. D3 Reconciliation with Data Quality Objectives Describe how the results obtained from the project or task will be reconciled with the results of the DQO Process. Describe how issues will be resolved. Discuss how limitations on the use of the data will be reported to decision makers. Identify the procedures used to assess precision, bias, and complete- ness for the project data. A methodology has been developed to assist users in reconciling data results with the DQOs. The Data Quality Assessment (DQA) Process*7* is used to assess the scientific and statistical quality of data collected for a specific purpose. In the DQA Process, the data will be analyzed scientifically to inspect for technical anomalies and to judge that the context of the data is correct. At the same time, the data will be evaluated statistically to confirm that the statistical model was correct by selecting a statistical test and validat- ing the test by verifying assumptions, such as for distribution and indepen- dence. The outcome of the DQA process will indicate whether a decision can be made using the existing data or additional data must be collected. 24 ------- APPENDIX A TERMS AND DEFINITIONS Activity - an all-inclusive term describing a specific set of operations or related tasks to be performed, either serially or in parallel (e.g., research and development, field sampling, analytical operations, equipment fabrication), that in total result in a product or service. Assessment - the evaluation process used to measure the performance or effectiveness of a system and its elements. In this Standard, assessment is an all-inclusive term used to denote any of the following: audit, performance evaluation, management systems review, peer review, inspection or surveillance. Audit - a planned and documented investigative evaluation of an item or process to determine the adequacy and effectiveness as well as compliance with established procedures, instructions, drawings, QAPPs, and other applicable documents. Characteristic - any property or attribute of a datum, item, process, or service that is distinct, describable, and measurable. Computer Program - a sequence of instructions suitable for processing by a computer. Processing may include the use of an assembler, a compiler, an interpreter, or a translator to prepare the program for execution. A computer program may be stored on magnetic media, and be referred to as "software", or may be stored permanently on computer chips, and be referred to as "firmware". Computer programs covered by this Standard are those used for design analysis, data acquisition, data reduction, data storage (data bases), operation or control, and data base or document control registers when used as the controlled source of quality information. Contractor - any organization or individual that contracts to furnish services or items or perform work. Corrective Action - measures taken to rectify conditions adverse to quality and, where necessary, to preclude their recurrence. Customer - any individual or organization for whom items or services are furnished or work performed in response to defined requirements and expectations. Data Quality Assessment (DQA) - a process of statistical and scientific evaluation that is used to assess the validity and performance of the data collection design and statistical test, and to establish whether a data set is adequate for its intended use. 25 ------- Data Quality Objectives (DQOs) - a statement of the precise data, the manner in which such data may be combined, and the acceptable uncertainty in those data in order to resolve an environmental problem or condition. This may also include the criteria or specifications needed to design a study that resolves the question or decision addressed by the DQO process. Data Quality Objectives Process - a Total Quality Management (TQM) tool, based on the Scientific Method and developed by the U.S. Environmental Protection Agency to facilitate the planning of environmental data collection activities. The DQO process enables planners to focus their planning efforts by specifying the use of the data (the decision), the decision criteria (action level), and the decision maker's acceptable decision error rates. The products of the DQO process are the DQOs (See also Graded Approach). Data Usability - the process of ensuring or determining whether the quality of the data produced meets the intended use of the data. Design Review - a documented evaluation by a team, including personnel other than the original designers, the responsible designers, the customer for the work or product being designed, and a QA representative to determine if a proposed design will meet the established design criteria and perform as expected when implemented. Engineered Environmental Systems - an all-inclusive term used to describe pollution control devices and systems, waste treatment processes and storage facilities, and site remediation technologies and their components that may be utilized to remove pollutants or contaminants from the environment. Examples include wet scrubbers (air), soil washing (soil), granulated activated carbon unit (water), and filtration (air, water). Usually, this term will apply to hardware-based systems; however, it will also apply to methods or techniques used for pollutant reduction or containment of contamination to prevent further movement of the contaminants, such as capping, solidification or vitrification, and biological treatment. Environmental Conditions - the description of a physical medium (e.g., air, water, soil, sediment) or biological system expressed in terms of its physical, chemical, radiological, or biological characteristics. Environmental Data - any measurements or information that describe environmental processes or conditions, or the performance of engineered environmental systems. Environmental Data Operations - work performed to obtain, use, or report information pertaining to environmental processes and conditions. Environmental Monitoring - the process of measuring or collecting environmental data. Environmental Processes - manufactured or natural processes that produce discharges to or 26 ------- impact the ambient environment. Environmental Programs - an all-inclusive term pertaining to any work or activities involving the environment, including but not limited to: characterization of environmental processes and conditions; environmental monitoring; environmental research and development; the design, construction, and operation of engineered environmental systems; and laboratory operations on environmental samples. Environmentally Related Measurements - the data collection activity or investigation involving the assessment of chemical, physical or biological factors in the environment which affect human health or the quality of life. Financial Assistance - the process by which funds are provided by one organization (usually government) to another organization for the purpose of performing work or furnishing services or items. Financial assistance mechanisms include grants, cooperative agreements, and government interagency agreements. Graded Approach - the process of basing the level of application of managerial controls applied to an item or work according to the intended use of results and the degree of confidence needed in the quality of the results. (See Data Quality Objectives Process). Hazardous Waste - any waste materials that satisfies the definition of "hazardous waste" as given in 40 CFR Part 261, "Identification and Listing of Hazardous Waste." Independent Assessment - an assessment performed by a qualified individual, group, or organization that is not a part of the organization directly performing and accountable for the work being assessed. Inspection - examination or measurement of an item or activity to verify conformance to specific requirements. Item - an all-inclusive term used in place of the following: appurtenance, facility, sample assembly, component, equipment, material, module, part, product, structure, subassembly, subsystem, system, unit, documented concepts, or data. Management - those individuals directly responsible and accountable for planning, implementing, and assessing work. Management System - a structured non-technical system describing the policies, objectives, principles, organizational authority, responsibilities, accountability, and implementation plan of an organization for conducting work and producing items and services. Management System Review (MSR) - the qualitative assessment of a data collection 27 ------- operation and/or organization (s) to establish whether the prevailing quality management structure, policies, practices, and procedures are adequate for ensuring that the type and quality of data needed are obtained. May - denotes permission but not a requirement. Method - a body of procedures and techniques for performing an activity (e.g., sampling, chemical analysis, quantification) systematically presented in the order in which they are to be executed. Mixed Waste - hazardous waste material, as defined by 40 CFR part 261 (RCRA), mixed with radioactive constituents. Must - denotes a requirement that has to be met. Peer Review - a documented critical review of work generally beyond the state of the art or characterized by the existence of potential uncertainty. The peer review is conducted by qualified individuals (or organization) who are independent of those who performed the work, but are collectively equivalent in technical expertise (i.e., peers) to those who performed the original work. The peer review is conducted to ensure that activities are technically adequate, competently performed, properly documented, and satisfy established technical and quality requirements. The peer review is an in-depth assessment of the assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance criteria, and conclusions pertaining to specific work and of the documentation that supports them. Peer reviews provide an evaluation of a subject where quantitative methods of analysis or measures of success are unavailable or undefined, such as in research and development. Performance Evaluation (PE) - a type of audit in which the quantitative data generated in a measurement system are obtained independently and compared with routinely obtained data to evaluate the proficiency of an analyst or laboratory. Procedure - a documented set of steps or actions that systematically specifies or describes how an activity is to be performed. Process - an orderly system of actions that are intended to achieve a desired end or result. Examples of processes include analysis, design, data collection, operation, fabrication, and calculation. QTRAK - is a Computer Program that contains database information on Quality Management Plans and Quality Assurance Project Plans to the Program Managers, Project Officers, and the OQA for planning and assessment of the status of regional Quality Management Plans and the associated Project Plans. 28 ------- Qualified Data - any data that have been modified or adjusted as part of statistical or mathematical evaluation, data validation, or data verification operations. Quality - the sum of features and properties/characteristics of a process, item, or service that bears on its ability to meet the stated needs and expectations of the user. Quality Assurance (QA) - an integrated system of management activities involving planning, implementation, assessment, reporting, and quality improvement to ensure that a process, item, or service is of the type and quality needed and expected by the customer. Quality Assurance Management Staff (QAMS) - the U.S. EPA's headquarters staff element that establishes and promulgates Quality Assurance Policy. Quality Assurance Officer (QAO) - the designated Region 6 staff member that has the delegated authority for approval of all Quality Management Plans in Region 6, Chief of the Office of Quality Assurance. Quality Assurance Program Description/Plan -see Quality Management Plan. Quality Assurance Project Plan (QAPP) - a formal document describing in comprehensive detail the necessary QA, QC, and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance criteria. Quality Control (QC) - the overall system of technical activities that measures the attributes and performance of a process, item, or service against defined standards to verify that they meet the stated requirements established by the customer. Quality Improvement - a management program for improving the quality of operations. Such management programs generally entail a formal mechanism for encouraging worker recommendations with timely management evaluation and feedback or implementation. Quality Indicators - measurable attributes of the attainment of the necessary quality for a particular environmental decision. Indicators of quality include precision, bias, completeness, representativeness, reproducibility, comparability, and statistical confidence. Quality Management - that aspect of the overall management system of the organization that determines and implements the quality policy. Quality management includes strategic planning, allocation of resources, and other systematic activities (e.g., planning, implementation, and assessment) pertaining to the quality system. Quality Management Plan (QMP) - a formal document that describes the quality system in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing all activities conducted. 29 ------- Quality System - a structured and documented management system describing the policies, objectives, principles, organizational authority, responsibilities accountability, and implementation plan of an organization for ensuring quality in its work processes, products (items), and services. The quality system provides the framework for planning, implementing, and assessing work performed by the organization and for carrying out required QA and QC. Radioactive Waste - waste material containing radionuclides, or contaminated by radionuclides. Readiness Review - a systematic, documented review of the readiness for the startup or continued use of a facility, process, or activity. Readiness reviews are typically conducted before proceeding beyond project milestones and prior to initiation of a major phase of work. Remediation - the process of reducing the concentration of a contaminant (or contaminants) in air, water, or soil media to a level that poses an acceptable risk to human health. Research (Applied) - a process, the objective of which is to gain knowledge or understanding necessary for determining the means by which a recognized and specific need may be met. Research (Basic) - a process, the objective of which is to gain knowledge or understanding of the fundamental aspect of phenomena and of observable facts without specific applications toward processes or products in mind. Research Development/Demonstration - Systematic use of the knowledge and understanding gained from research and directed toward the production of useful materials, devices, systems, or methods, including prototypes and processes. Self-Assessment - Assessments of work conducting by individuals, groups, or organizations directly responsible for overseeing and/or performing the work. Service - the category of economic activity that does not produce manufactured items. In environmental data operations or engineering projects, such activities include design, inspection, laboratory and/or field analysis, repair, and installation. Shall - denotes a requirement that is mandatory and has to be met. Should - denotes a guideline or recommendation. Significant Condition - any state, status, incident, or situation of an environmental process or condition of an engineered environmental system in which the work being performed will be adversely affected in a manner sufficiently serious to require corrective action to satisfy quality objectives or specifications and safety requirements. 30 ------- Software Life Cycle - the period of time that starts when a software product is conceived and ends when the software product is no longer available for routine use. The software life cycle typically includes a requirements phase, a design phase, an implementation phase, a test phase, an installation and check-out phase, an operation and maintenance phase, and sometimes a retirement phase. Standard Operating Procedure (SOP) - a written document that details the method for an operation, analysis, or action with thoroughly prescribed techniques and steps, and that is Officially approved as the method for performing certain routine or repetitive tasks. Supplier - any individual or organization furnishing items or services or performing work according to a procurement document or financial assistance agreement. This is an all- inclusive term used in place of any of the following: vendor, seller, contractor, subcontractor, fabricator, or consultant. Surveillance - the act of monitoring or observing a process or activity to verify conformance to specified requirements. Technical Review - a documented critical review of work that has been performed within the state of the art. The review is accomplished by one or more qualified reviewers who are independent of those who performed the work, but are collectively equivalent in technical expertise to those who performed the original work. The reviews are an in-depth analysis and evaluation of documents, activities, material, data, or items that require technical verification or validation for applicability, correctness, adequacy, completeness, and assurance that established requirements are satisfied. Technical Systems Audit (TSA) - a thorough, systematic, on-site qualitative audit of facilities, equipment, personnel, training procedures, record keeping, data validation, data management, and reporting aspects of a system. Total Quality Management (TQM) - the process of applying quality management to all activities of the organization, including technical and administrative operations. See Quality Management and Quality System. Validation - an activity that demonstrates or confirms that a process, item, data set, or service satisfies the requirements defined by the user. Verification - the act of authenticating or formally asserting the truth that a process, item, data set, or service is, in fact, that which is claimed. Work - the process of performing a defined task or activity (e.g., research and development, field sampling, analytical operations, equipment fabrication). 31 ------- |