United States National Center for Environmental Research
Environmental Protection Office of Research and Development
Agency Washington, DC 20460
Guidance on Satisfying EPA Quality
Requirements for STAR Grants
NCER G-l STAR
Revised June 22, 2007
-------
FOREWORD
The Environmental Protection Agency's National Center for Environmental Research's Science
to Achieve Results or STAR program funds research grants and graduate fellowships in
numerous environmental science and engineering disciplines through a competitive solicitation
process and independent peer review. The program engages the nation's best scientists and
engineers in targeted research that complements EPA's own outstanding intramural research
program and those of our partners in other federal agencies. In addition, through this same
competitive process, NCER periodically establishes large research centers in specific areas of
national concern. The most recent information about STAR programs, including grant
opportunity announcements, can be found on the NCER web site:
http V/www. epa. gov/ncer
This document assists potential applicants and grantees in meeting the EPA's quality
requirements for the Science to Achieve Results (STAR) Program, both before and after award.
It provides information for you, the applicant, in documenting the quality assurance approach to
be used in the proposal and after award when the award document requires more detailed
information is needed. Quality assurance documentation for the STAR Program includes
Quality Assurance Statements (Section 2), Quality Assurance Project Plans (Section 3), and
Quality Management Plans (Section 4).
-------
TABLE OF CONTENTS
FOREWORD i
1. BACKGROUND 1
1.1 Why is Quality Assurance Necessary for Research Grants? 1
1.2 What are EPA's Quality Assurance Requirements for Research Grants? 2
1.3 What are EPA's Quality Assurance Requirements for STAR Research Grants? 3
2. QUALITY ASSURANCE STATEMENT 5
3. QUALITY ASSURANCE PROJECT PLAN 8
4. QUALITY MANAGEMENT PLAN 15
5. REPORTS AND IMPLEMENTATION 17
11
-------
Guidance on Satisfying
EPA Quality Requirements for STAR Grants
1. BACKGROUND
1.1 Why is Quality Assurance Necessary for Research Grants?
Quality assurance (QA) and quality control (QC)1 practices provide prudent safeguards against
the occurrence of problems and the introduction of error into the data produced which could
adversely impact the results and the conclusions made from them. EPA believes that as a
potential grant applicant considers the approaches to be taken to successfully complete the
research project and obtain documented quality data, the applicant should also identify and
document the activities that will ensure that the product of the research is of adequate quality to
be used as planned. Such documentation is beneficial when a grant application is peer reviewed
or a manuscript is submitted for publication. Documents demonstrating that the data meet
applicable and appropriate quality standards or criteria are critical and are positively received by
many reviewers.
A well-written quality plan may also help detect problems or incorrect assumptions before work
begins and, thus, avoid false starts or generation of questionable or unusable data sets. Results of
a well-designed experiment can be invalidated by simple things such as misunderstanding verbal
directions that could have been included in standard operating procedures (SOPs) or research
protocols.2 Similarly, an improperly calibrated sensor, such as a pH electrode, can result in data
gathered at great expense of time and money being unusable by the researchers. Finally, many
research projects are so novel that the chances of failure are higher than for other types of
research. Developing a framework for the early detection of errors and for the documentation of
the steps in the experiment will help to assure the reproducibility of the research and lead to
successful completion of the project with data of known quality.
The EPA believes strongly in the application of QA and QC principles to all types of
environmental studies. In science and engineering research studies, products may range from a
clear-cut identification of a fundamental photochemical reaction mechanism (which may lead to
a better understanding of ozone formation) to a realistic assessment of the efficiency and
longevity of a novel catalytic converter for reducing automobile tailpipe emissions. To ensure
that such products are of sufficient and adequate quality for their intended use, the application of
QA and QC is both prudent and necessary regardless of the level of complexity of the work.
1 EPA Order 5360.1 defines Quality assurance (QA) is an integrated system of management activities involving planning, implementation,
documentation, assessment, reporting, and quality involvement to ensure that a process, item, or service is of the type and quality needed and
expected by the customer and Quality control (QC) is the overall system of technical activities that measures the attributes and performance of a
process, item, or service against defined standards to verify that they meet the stated requirements established by the customer; the operational
techniques and activities used to fulfill quality requirements.
2 In this context, SOPs are defined as written and officially approved documents that detail the preferred method for performing an operation such
as sample collection, analysis, equipment use or other routine tasks with thoroughly prescribed techniques and steps to ensure consistency.
Research protocols are written directions that describe the plan of a scientific experiment such as the experimental design and data collection
activities, and they may be adapted or modified as the experiment progresses or when an experiment is added to the project.
1
-------
Accordingly, EPA has established quality requirements that must be followed within EPA and by
extramural contractors and financial assistance recipients for all work performed that involves
environmental data collection, use or reporting (including both primary and secondary data).
1.2 What are EPA's Quality Assurance Requirements for Research Grants?
EPA assistance agreement recipients that conduct environmental programs must implement or
have implemented a quality program that conforms to the American National Standard
ANSI/ASQC E4-1994, Specifications and Guidelines for Quality Systems for Environmental
Data Collection and Environmental Technology Programs (ASQ, 1994). This requirement is
defined in 40 Code of Federal Regulations (CFR) Part 30, Grants and Agreements with
Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations, as applied to
the STAR program (Parts 30.45 and 30.54). Any awards made to State, Tribal, or local
governments would be covered by quality requirements defined in 40 CFR Part 31, Uniform
Administrative Requirements for Grants and Cooperative Agreement to State, Tribal, and Local
Governments.
EPA's quality requirements apply to projects that include:
data collection or processing, including the use of secondary data;
conducting surveys;
making environmental measurements;
describing processes or conditions;
describing ecological or health effect and consequences
creating or modifying models; and
developing environmental technology (whether hardware-based or via new techniques
and methods) for pollution control and waste treatment.
Customarily, the EPA requirements for extramural agreements include:
1. documentation of the organization's quality program (usually in a Quality Management
Plan) and
2. documentation of quality assurance and quality control activities to specific efforts
(usually in a QA Project Plan).
The level of detail needed to document quality practices depends on the type of work and the
intended use of the results. For more information on EPA's quality program, see
http://www.epa.gov/quality.
-------
1.3 What are EPA's Quality Assurance Requirements for STAR Research
Grants?
Research is often conducted simply to develop information rather than to be used in decision
making, with peer-reviewed publications as a result. As a result, the STAR program has tailored
the customary EPA quality documentation to allow more flexibility. Potential quality
documentation for STAR research grant application includes:
QA Statement (Section 2): A brief description of the QA and QC activities for a specific
project and identifies the individual responsible for ensuring these activities are
performed. A QA Statement is typically requested for all STAR grant applications
(usually in the RFA), and is usually provided by the applicant as part of the proposal.
This documentation usually satisfies EPA requirements (1) and (2) in Section 1.2.
QA Project Plan (Section 3): A detailed description of the QA and QC activities for a
specific project. After an award decision is made, a QA Project Plan may be requested
as part of the terms and conditions of an award to supplement the QA Statement.
This documentation satisfies EPA requirement (2) in Section 1.2.
Quality Management Plan (Section 4): A description of an organization's quality
program. A Quality Management Plan will be requested as part of the terms and
conditions of awards for Research Centers or for STAR grants that are extremely large
or have direct regulatory impacts. This documentation satisfies EPA requirement (1) in
Section 1.2. Research Centers then develop their own procedures for the development,
review, and approval of documentation to satisfy EPA requirement (2) in Section 1.2.
The purpose of these documents is to provide information to the EPA Project Officer, Quality
Assurance Manager, peer reviewers, and award officials on an applicant's capabilities to provide
minimum required quality assurance and quality control for the proposed work. These
documents should demonstrate that QC procedures are in place to ensure that each project is
successfully completed and the objective is achieved. Assistance recipients are encouraged to
create written standard operating procedures (SOPs) and protocols, and reference them in these
documents.
-------
-------
2. QUALITY ASSURANCE STATEMENT
A Quality Assurance Statement (QAS) is a brief description of the quality assurance and quality
control practices that will be applied during a research project to assure that the results obtained
satisfy the project objectives. The majority of the Requests for Applications require that the
grant applicant provide this statement as part of the proposal. The QAS usually does not exceed
three pages and should either include the requested information, reference (by page and
paragraph number) the specific relevant portion of the research plan containing the information,
or provide a clear logical justification as to why the item does not apply to the proposed research.
The elements of a QA Statement are:
Quality Assurance Statement (1 to 3 pages in addition to the 15-page research plan)
For projects involving environmental data collection or processing, conducting surveys,
modeling, method development, or the development of environmental technology (whether
hardware-based or via new techniques), provide a Quality Assurance Statement (QAS) regarding
the plans for processes that will be used to ensure that the products of the research satisfy the
intended project objectives. Follow the guidelines provided below to ensure that the QAS
describes a system that complies with ANSI/ASQC E4, Specifications and Guidelines for
Quality Systems for Environmental Data Collection and Environmental Technology Programs.
Do not exceed three consecutively numbered, 8.5x11-inch pages of single-spaced, standard 12-
point type with 1-inch margins.
NOTE: If selected for award, applicants will be expected to provide additional quality
assurance documentation if requested by the terms and conditions of the agreement.
Address each applicable section below by addressing the following points or referencing the
specific location of the information in the Research Plan. (Not all will apply.)
(1) Identify the individual who will be responsible for the quality assurance (QA) and
quality control (QC) aspects of the research along with a brief description of this person's
functions, experience, and authority within the research organization. Describe the
organization's general approach for conducting quality research. (QA is a system of
management activities to ensure that a process or item is of the type and quality needed for
the project. QC is a system of activities that measures the attributes and performance of a
process or item against the standards defined in the project documentation to verify that
they meet those stated requirements.)
(2) Discuss project objectives, including quality objectives, any hypotheses to be tested,
and the quantitative and/or qualitative procedures that will be used to evaluate the success
of the project. Include any plans for peer or other reviews of the study design or analytical
methods.
-------
(3) Address each of the following project elements as applicable:
(a) Collection of new/primary data:
(Note: In this case the word "sample " is intended to mean any finite part of a statistical
population whose properties are studied to gain information about the whole. If certain
attributes listed below do not apply to the type of samples to be used in your research, simply
explain why those attributes are not applicable.)
(i) Discuss the plan for sample collection and analysis. As applicable, include sample
type(s), frequency, locations, sample sizes, sampling procedures, and the criteria for
determining acceptable data quality (e.g., precision, accuracy, representativeness,
completeness, comparability, or data quality objectives).
(ii) Describe the procedures for the handling and custody of samples including sample
collection, identification, preservation, transportation, and storage, and how the
accuracy of test measurements will be verified.
(iii)Describe or reference each analytical method to be used, any QA or QC checks or
procedures with the associated acceptance criteria, and any procedures that will be
used in the calibration and performance evaluation of the analytical instrumentation.
(iv)Discuss the procedures for overall data reduction, analysis, and reporting. Include a
description of all statistical methods to make inferences and conclusions, acceptable
error rates and/or power, and any statistical software to be used.
(b) Use of existing/secondary data (i.e., data previously collected for other purposes or
from other sources):
(i) Identify the types of secondary data needed to satisfy the project objectives.
Specify requirements relating to the type of data, the age of data, geographical
representation, temporal representation, and technological representation, as
applicable.
(ii) Specify the source(s) of the secondary data discuss the rational for selection.
(iii)Establish a plan to identify the sources of the secondary data in all
deliverables/products.
(iv)Specify quality requirements and discuss the appropriateness for their intended
use. Accuracy, precision, representativeness, completeness, and comparability
need to be addressed, if applicable.
(v) Describe the procedures for determining the quality of the secondary data.
(vi)Describe the plan for data management/integrity.
(c) Method development:
(Note: The data collected for use in method development or evaluation should be described
in the QAS as per the guidance in section 3A and/or 3B above.)
Describe the scope and application of the method, any tests (and measurements) to be
conducted to support the method development, the type of instrumentation that will
be used and any required instrument conditions (e.g., calibration frequency), planned
QC checks and associated criteria (e.g., spikes, replicates, blanks), and tests to verify
-------
the method's performance.
(d) Development or refinement of models:
(Note: The data collected for use in the development or refinement of models should be
described in the QAS as per the guidance in section 3A and/or 3B above.)
(i) Discuss the scope and purpose of the model, key assumptions to be made during
development/refinement, requirements for code development, and how the model
will be documented.
(ii) Discuss verification techniques to ensure the source code implements the model
correctly.
(iii)Discuss validation techniques to determine that the model (assumptions and
algorithms) captures the essential phenomena with adequate fidelity.
(iv)Discuss plans for long-term maintenance of the model and associated data.
(e) Development or operation of environmental technology:
(Note: The data collected for use in the development or evaluation of the technology should
be described in the QAS as per the guidance in section 3A and/or 3B above.)
(i) Describe the overall purpose and anticipated impact of the technology.
(ii) Describe the technical and quality specifications of each technology component or
process that is to be designed, fabricated, constructed, and/or operated.
(iii)Discuss the procedure to be used for documenting and controlling design changes.
(iv)Discuss the procedure to be used for documenting the acceptability of processes
and components, and discuss how the technology will be benchmarked and its
effectiveness determined.
(v) Discuss the documentation requirements for operating instructions/guides for
maintenance and use of the system(s) and/or process(s).
(f) Conducting surveys:
(Note: The data to be collected in the survey and any supporting data should be described in
the QAS as per the guidance in section 3A and/or 3B above.)
Discuss the justification for the size of the proposed sample for both the overall
project and all sub samples for specific treatments or tests. Identify and explain the
rational for the proposed statistical techniques (e.g., evaluation of statistical power).
4.) Discuss data management activities (e.g., record-keeping procedures, data-handling
procedures, and the approach used for data storage and retrieval on electronic media).
Include any required computer hardware and software and address any specific
performance requirements for the hardware/software configuration used.
-------
3. QUALITY ASSURANCE PROJECT PLAN
In some cases, the QA Statement may not contain sufficient quality assurance and quality control
information for a particular application and EPA may require that a more comprehensive QA
Project Plan be approved before you begin your research activities. The QA Project Plan
typically may be required for studies producing large volumes of data, determined to be
controversial by EPA, of a highly complex nature that may need more extensive documentation
of the planning process, or be significant in terms of the impact of or funding for the work. In
this case, a term and condition statement will be added to the award document which specifies
that more documentation is necessary, and that work involving environmental data generation
may not begin until the EPA Project Officer or Quality Assurance Manager provides you written
notification that the quality assurance plan is approved.
The terms and condition statements in the agreement will indicate whether the QA Project Plan
should be prepared in accordance with guidance that will be provided by the EPA Project Officer
or in accordance with the full EPA Requirements for Quality Assurance Project Plans (EPA,
2001) (available at http ://www. epa. gov/qualitv/qs-docs/r5 -final. pdf) and EPA's Guidance for
Quality Assurance Project Plans (EPA, 2002) (available at http ://www. epa. gov/quality/qs-
docs/g5-final.pdf).
The QA Project Plan should describe the QC and QA practices to be implemented by the
applicant for the proposed project in sufficient detail to present a clear picture of what is to be
done and when. Before writing your QA Project Plan, check with the EPA Project Officer to
determine which elements are applicable for your grant. In addition to the examples and
guidance found on NCER's website, additional information on QA Project Plans, including
examples and answers to frequently asked questions, is contained at epa.gov/quality/qapps.html.
Although guidance for writing a QA Project Plan may be customized for a specific project by the
EPA Project Officer and the NCER Quality Assurance Manager, the guidance that they will
provide if indicated by the terms and conditions of the agreement is likely to resemble the
following:
-------
Quality Assurance Project Plans (QAPPs)
General Guidance for Writing and Reviewing QAPPs
For EPA/NCER and the STAR Grant Program
General Information about QAPPs -
The purpose for writing a QAPP is to ensure that the activities associated with the collection,
generation, use, and/or reporting of data will provide information suitable for answering a
question or making a decision, such as:
a hypothesis is correct, or not;
a method works, or not;
a new technology is beneficial, and it is "this much" better than the old standard;
this model can tell us "XYZ", but with these limitations.
Basically, state what you are going to do, how you are going to do it, and how you will know
that you have done it "right". In the QAPP, you will define what "right" is for your research.
For example, when you provide specific criteria for some measure of accuracy, then also explain
the reasons for needing that level of accuracy (or inaccuracy, as the case may be). The goal is to
be specific enough about how you will conduct the research that it could be reproduced by
another team of researchers with reasonably similar results. Assume that the reader has already
read your proposal, and refer to specific sections of it as necessary.
The QAPP must cover all areas / phases of your research, so ensure that the plan is complete. If
necessary, break it into logical sections for different activities. If the details of later phases
depend on information gleaned in the early ones, provide the information you have now, and
provide your EPA PO with an addendum with the remaining information at a later date.
However, all research activities must be covered under a QAPP approved by your EPA PO
before you can begin working on those activities. So ensure timely submission of your plans for
the next phase. Failure to obtain EPA approval on the requisite documentation in a timely
manner may result in a freeze on your funding.
QAPPs must be written in active language indicating exactly what will be done during the course
of the project and not in terms of "may", "should", or "could" as might have been done during
the proposal stage of the research. Now that the research has been funded, write in concrete
terms of what will be done with the money. The QAPP must be significantly more detailed than
the Quality Assurance Statement that was submitted with the proposal. There is no page limit;
but short, to the point, yet complete is preferred.
This summary guidance provides information on the general content to be provided in QAPPs,
but it is not a requirement to follow the format described below. It is far more important to
cover all of the required details in a format that makes sense for the type of research project in
question. The EPA QA/R-5 document (referenced below) provides a generally accepted,
formalized structure that many people are already familiar with and may wish to use.
9
-------
This information is a short summary of EPA's "Guidance for Quality Assurance Project Plans
(EPA QA/G-5)" (http://www.epa.gov/quality/qs-docs/g5-fmal.pdf) and "EPA Requirements for
Quality Assurance Project Plans (EPA QA/R-5)" (http://www.epa.gov/qualitv/qs-docs/r5-
final.pdf). If necessary, additional details can be found within these online documents.
The first part of this guide present general guidance that would be applicable to almost any type
of research. Then the following section covers additional issues that are specific to certain types
of data and/or specific types of research (e.g. modeling, technology development, analytical
method development, surveys, etc.). One or more of the issues discussed may apply to any given
project, but it is not necessary to read the ones that obviously do not apply.
Parts of a QAPP are:
Background/purpose: Lay out the question to be answered and/or decision to be made.
Experimental Design: Describe what will be done.
Data Gathering Methods: Connect what will be done with how it will be done.
Data Quality: Show how we can be sure it will be done well enough (acceptance criteria).
Data Reduction: Discuss the procedures that will make the data meaningful.
Interaction of the Players: Show the responsibilities of each organization / researcher.
Each of these segments is discussed on the following pages.
NOTE: It is not necessary to repeat information that is in the Research Plan (RP) that was
submitted with the proposal. Simply make reference to pages in the RP as necessary.
Background/Purpose - Lay out the question to be answered and/or decision to be made.
State explicitly the question to be answered and/or the decision to be made. (This must
be clear if we are to evaluate the usefulness of methodologies.) Indicate the reasons why
this research is important.
Describe how the research group will obtain the data necessary to provide the
information neededio answer the question or make the decision, so that the Project
objectives are connected with the Data objectives, (i.e., how will the methods be used to
get you the data you need?) Show what general kind of information is needed (e.g.
sampling, monitoring, analysis, compilation of secondary data, etc.)
10
-------
Design - Describe what will be done.
Provide a summary of all technical work to be performed and/or products to be produced.
Describe the connections between your stated purpose and what will be done to
successfully achieve it. [AND define the criteria for "success" for the research].
Describe the scheme for different types of data collection (or compilation) activities to be
used (include conditions and assumptions). Explain the reasons for designing the project
in such a way, and provide explanation for any quality control (QC) criteria you set
in this section.
Show how this scheme will produce the group of results needed to answer the question,
etc. from above. (This is essentially the Data Quality Objectives (DQO) Process, which
is a QA system name for a set of steps that mirror the scientific method. The bigger the
project, the more important it is to utilize true DQOs in developing the QAPP. For
additional information see http://www.epa.gov/quality/qs-docs/g4-fmal.pdf)
Data Gathering Methods - Connect what will be done with how it will be done.
(Includes analytical work, secondary data, meta-data, surveys, modeling efforts, etc.)
Describe what specific types of data need to be gathered, how they will be gathered or
produced, and how each one connects with and/or supports the stated purpose and criteria
for success of the research.
Describe the specific methods and equipment to be used. If established methods are to be
used cite them and attach copies if they are not readily available. (Use of tables may be
helpful.)
Include operating procedures for major analytical work and/or unique research
procedures, and provide a table listing all other repetitive operating procedures that are
followed in the process of the research. (It is important to show that the research is
repeatable.)
Data Quality - Show how we can be sure it will be done well enough (acceptance criteria).
Data quality encompasses both Quality Control (QC) and Quality Assurance (QA)
practices. QC is the system of technical activities used in data gathering
activities/methods to ensure the quality of each individual data point (as applicable:
precision, accuracy, representativeness, completeness, & comparability); and Quality
Assurance (QA) is the overall program that includes planning, QC, documentation, and
Assessments or audits.
11
-------
List the QC, QA, and Assessment activities that will be part of the project, to show how
the required data quality will be achieved. (What will the numbers represent and what
will we do with them?)
Explain the reasons for the research design, and provide explanation for any quality
control (QC) criteria you set in this section.
For each activity, state appropriate acceptance limits/criteria. Include all QC parameters
for each type of laboratory analysis, secondary data sources, model validation, statistical
procedures, etc. Indicate how each will be documented and what corrective actions will
be taken if QC criteria are not met. (Again, use of tables may be helpful.)
For each QC parameter (Precision, Accuracy, Representativeness, Completeness,
Comparability), state the how conformance with the stated limits will be determined, and
the consequences of failure such as for precision, what is the acceptable range, how will
it be determined (average of 3 lab duplicates prior to the start, etc.), if fail, recalibrate and
repeat until satisfactory, etc. Who makes the decisions if some parameters fail?
For each assessment type (field audit, lab audit, data quality assessment, etc.), state the
type and acceptance level, if any, and the consequences of failure.
Of all of the data quality parameters (Precision, Accuracy, Representativeness,
Completeness, Comparability), the most important is Representativeness. Describe the
representativeness of the sample(s) in the plan. How is the statistical network designed?
(If the research does not use representative data, it doesn't matter how good it is, or how
good the use of it is. This is the main connection between the science and the politics:
making sure that the right samples will be collected to validly address the issue at hand.)
Description of Data Reduction Methods and Procedures - Discuss the procedures that will
make the data meaningful.
Describe all data reduction methods and procedures.
Identify the specific descriptive statistical methods (for example, regression analyses,
analysis of variance, or multivariate analyses) that will be used to present results.
Discuss how raw and processed data that are used in statistical analyses will be verified
after statistical analyses have been completed. Also include a complete citation of
software programs that will be used for these statistical analyses and to present results.
When using secondary data, include a discussion of the quality of these data and how the
data will be transferred into computer files for various analyses and how they are verified
through these processes. Discuss how the quality of these secondary data affect the
results being reported. Indicate that a complete citation of these data sources will be
complied during data collection so that these data can be reviewed later if necessary.
12
-------
Discuss how original or raw data measurements will be verified after they have been
transferred from instrument data recording devices and/or floppy diskettes or laboratory
notebooks and processed by computer or manually.
Interaction of the Players - Show the responsibilities of each organization / researcher.
Need clear organizational charts for the project, showing who is responsible for what,
and who reports to whom. This must include every organization involved in the project:
management, project management including co-Pis, field groups, lab workers, QA,
contractors, consultants. Indicate how each party will interact to achieve the end result.
Plan must include statements of who has the authority to change it or the operating
procedures.
It should be signed by a representative of each organization to confirm their involvement,
understanding, and acceptance of the stated roles, responsibilities, and authorities. (A
common flaw is that one group writes and signs the Plan and sends it to EPA without the
concurrence of all parties.)
Special Topics:
GIS/Remote Sensing data: Discuss how the following elements will be addressed: positional
accuracy; attribute accuracy; logical consistency; time; lineage; resolution accuracy; and the
completeness of coverage, classification, and verification.
Conducting surveys: Discuss the justification for the size of the proposed sample for both the
overall project and all subsamples for specific treatments or tests. Identify and explain the
rational for the proposed statistical techniques (e.g., evaluation of statistical power).
Development or refinement of models:
(i) Discuss the scope and purpose of the model, key assumptions to be made during
development/refinement, requirements for code development, and how the model will
be documented.
(ii) Discuss verification techniques to ensure the source code implements the model
correctly.
(iii) Discuss validation techniques to determine that the model (assumptions and algorithms)
captures the essential phenomena with adequate fidelity.
(iv) Discuss plans for long-term maintenance of the model and associated data.
13
-------
Development or operation of environmental technology:
(i) Describe the overall purpose and anticipated impact of the technology.
(ii) Describe the technical and quality specifications of each technology component or
process that is to be designed, fabricated, constructed, and/or operated.
(iii) Discuss the procedure to be used for documenting and controlling design changes.
(iv) Discuss the procedure to be used for documenting the acceptability of processes and
components, and discuss how the technology will be benchmarked and its effectiveness
determined.
(v) Discuss the documentation requirements for operating instructions/guides for
maintenance and use of the system(s) and/or process(s).
Secondary Data Collection:
(Secondary data is data that will be used for purposes other than those for which they were
originally collected. They may be obtained from many sources, including literature, industry
surveys, compilations from computerized databases and information systems, and computerized
or mathematical models of environmental processes. This includes purchased data sets, too.)
(i) Identify the types of secondary data needed to satisfy the project objectives. Specify
requirements relating to the type of data, the age of data, geographical representation,
temporal representation, and technological representation, as applicable.
(ii) Specify the source(s) of the secondary data discuss the rational for selection.
(iii) Establish a plan to identify the sources of the secondary data in all
deliverables/products.
(iv) Specify quality requirements and discuss the appropriateness for their intended use.
Accuracy, precision, representativeness, completeness, and comparability need to be
addressed, if applicable.
(v) Describe the procedures for determining the quality of the secondary data.
(vi) If no quality documentation exists or if the quality of the secondary data will not be
evaluated under the grant, the QAPP shall require that a disclaimer be added to all
deliverables/products (including models and other decision support tools). The
wording for the disclaimers shall be defined. (Use of disclaimers must receive prior
approval from the EPA Project Officer.)
(vii) Describe all data reduction procedures including calculations, equations, scripts, and
statistical analysis. At some point, relate these back to satisfying the project objectives.
(viii) Include a plan for data management / integrity.
14
-------
4. QUALITY MANAGEMENT PLAN
Quality Management Plans (QMPs) are typically required for Research Centers and other grants
with multiple distinct projects. A QMP is a document that describes an organization/program's
quality system in terms of the organizational structure, policy and procedures, functional
responsibilities of management and staff, lines of authority, and required interfaces for those
planning, implementing, documenting, and assessing all activities conducted across multiple
projects. [772/5 is in contrast to project-specific documentation, such as a Quality Assurance
Project Plan (QAPP), that describes the necessary QA, QC, and other technical activities that
must be implemented to ensure that the results of the work performed will satisfy the stated
performance criteria.} The elements of a QMP are described below. Your QMP must be signed
and dated by senior manager, senior line management, and QA Manager. Additional information
on QMPs, including examples and answers to frequently asked questions, is contained at
epa.gov/quality/qmps.html3.
(1) Management and Organization: State the organization's QA policy and how
management assures that all programs and Pis understand and implement QA and QC
activities. Identify all components of organization, the position of QA Manager, and the
lines of reporting of the QA Manager. (This may be done through an organizational
chart.) Discuss the authorities of the QA Manager and staff and demonstrate that the QA
Manager is both qualified and independent of data collection or use activities. Discusses
technical activities or programs that require quality management and where internal
coordination of QA and QC activities among organizations or co-Pis is needed. The
Center QA Manager cannot be one of the co-Pis or research personnel for the
projects covered by scope of the agreement.
(2) Quality System Components: Describes principal quality components (e.g., quality
program documentation, annual reviews, project-specific quality documentation) along
with the responsibilities of management and staff for each component. The Center QMP
must indicate the requirement for each project to develop a Quality Assurance Project Plan
(QAPP) that is compliant with "EPA Requirements for Quality Assurance Project Plans
(EPA QA/R-5)" {http://www. epa.gov/quality/qs-docs/r5-final.pdf}. Additional guidance
for developing QAPPs can be found at http://www.epa.gov/qualitv/qs-docs/g5-final.pdf.
The QMP must also state the following requirements: 1) any type of planned work with
environmental data may not begin until the QAPP has been written and 2) the QAPP has
been reviewed and approved for completeness and compliance with the R-5 document by
the Center's QA Manager. The QMP must describe the process for submission, review,
documentation of approval, revision control, distribution, and implementation of QAPPs.
3Note: The STAR program has tailored the Agency elements to meet the needs of this grant program. Therefore, all the information contained on
this web page may not be applicable to your grant.
15
-------
(3) Qualifications and Training: Describe the process for ensuring and documenting that
personnel have necessary quality-related qualifications and training.
(4) Procurement of Items and Services: Describe the process for review and approval of
extramural agreements (e.g., grants and contracts) and responses to solicitations to ensure
that they satisfy all technical and quality requirements. Describe the process to ensure the
quality or acceptability of Suppliers' products.
(5) Documents and Records: Describe the process for preparing, reviewing, approving,
issuing, using, and revising documents and records. Describe the process for maintaining
documents and records including retention, access, preservation, traceability, removal of
obsolete documentation, and disposition.
(6) Computer Hardware and Software: Describe the process for developing, installing,
testing, using, maintaining, controlling, and documenting computer hardware and
software.
(7) Planning (This element is particularly important for Research Centers): Describe the
process for developing, reviewing, approving, implementing and revising project-level
QA and QC documentation. This documentation should include: project goals, objectives,
and questions to be addressed; the project schedule, resources, and milestones; the type
and quantity of data needed and how the data will be used to support the project's
objectives; performance criteria for measuring quality; QA and QC activities to assess the
performance criteria; and a description of how, when, and where the data will be obtained
(including secondary/existing data) and identification of any constraints on data collection.
(8) Implementation of Work Processes: Describe the process for ensuring that work is
performed according to planning and technical documents (e.g., project narrative,
Standard Operating Procedures, etc.). Describe the process for identifying operations
which need standard procedures for uniformity (e.g., SOPs) and the process for preparing,
reviewing, approving, revising, and withdrawing these procedures.
(9) Assessment, Response, and Improvement: Describe the process for reviewing the
organization's quality program, at least annually. Describe the process for planning,
implementing and documenting assessments of QA/QC across projects, and how the
findings of these assessments will be addressed by management. Describe the process for
ensuring that conditions adverse to quality are identified and promptly corrected and/or
prevented. The Center QA Manager/Officer must conduct Technical Systems
Assessments (TSAs) on-site for each project during the first year research is being
performed and at least every-other year thereafter. On the years in-between on-site TSAs,
the QA Manager/Officer must go through the TSA checklist by conference call. TSA
reports and completed checklists must be maintained as part of the Center's records. The
QMP must also describe procedures for Data Quality Assessments and any other types of
16
-------
assessments/audits to be conducted. EPA guidance on different types of assessments
(including TSAs) can be found at: http://www.epa.gov/quality/qs-docs/g7-fmal.pdf
5. REPORTS AND IMPLEMENTATION
While summaries of QA and QC results may be included as an appendix to the final report,
specific QA and QC text descriptions are required by your grant's terms and conditions to be
addressed as part of annual and final reports. We suggest you contact your sponsored
programs/business office to obtain a copy of the terms and conditions, as they are legally binding
on a recipient. In your reports, be sure to note your laboratory's QA and QC plans, assessment
activities, and any calibration and verification services performed outside your research unit (for
example, verification of a microbalance, calibration of a flow meter, or tuning of a
spectrophotometer). Also, any secondary data used in the project must have a complete citation.
Following the grant award and approval of the QAS, QA Project Plan, and/or Quality
Management Plan, (if required by the terms and conditions of the agreement) EPA expects that
the elements of the applicable document will be implemented as part of the agreement. If you
need to alter your approach as the research progresses to improve the integrity of the work or the
validity of the results, these changes should be discussed in your annual and final reports.
17
------- |