UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
RESEARCH TRIANGLE PARK, NC 27711
OFFICE OF AIR QUALITY PLANNING AND STANDARDS
Steps to Qualify or Validate Data after an Exceedance of Critical Criteria Checks
1/30/2018
NOTE: This memo, originally posted on AMTIC on 8/30/20 17, has been revised only to
replace the qualifier "IC" with the qualifier "1C". When the memo was originally written
EPA did not notice that a "IC" qualifier had already been defined in AQS.
In order to address issues related to the recent OIG Management Alert1 associated with findings of
failed 1-point quality control (QC) checks and data invalidation, EPA is providing some additional
guidance on the process to validate or invalidate routine data based on an exceedance of important
checks that have been identified as "critical criteria" in the QA Handbook2. These critical criteria checks3
are part of a validation template that were developed for all criteria pollutants around 2006 by EPA and
the monitoring organizations. Monitoring organizations, in their organizations specific quality assurance
project plans, may identify additional checks that they deem critical. The definition of the critical criteria
can be found in Appendix D of the QA Handbook but the following quote is the driver behind this
guidance:
"Observations that do not meet each and every criterion on the Critical Criteria should be
invalidated unless there are compelling reasons and justification for not doing so."
Compelling evidence (reason) is data, such as (but not limited to) an independent audit point(s), a multi-
point verification, and/or a prior zero/span check that establishes whether the analyzer was in fact
operating within the percent difference critical criteria acceptance limits and whether the 1-point QC
check itself is considered valid or invalid. This evidence is either available from routine tests within the
timeframe of the last acceptable valid QC check or from an independent test/check that establishes that
the system that produced the 1-QC was invalid once the failure is discovered. Because these are critical
criteria, timely action (i.e., the next day) on the part of the monitoring organization to determine why
such a "critical" failure occurred should be the normal course of action.
We define valid and invalid QC checks as follows:
• A valid QC check is one that is conducted using certified, properly functioning equipment,
conducted in a manner that adheres to appropriate procedures (SOPs) and the test
concentration is accepted as accurate.
# %
£> dnt *
sis
s
%
4> r-<
*1 PR0^
¦>
1 Report: Certain State, Local and Tribal Data Processing Practices Could Impact Suitability of Data for 8-Hour
Ozone Air Quality Determinations https://www.epa,goy/office-inspector-general/report-certain-state-tocal-and--
tribal-data-processine-practices-coyld
2 Quality Assurance Handbook for Air Pollution Measurement Systems Volume II Ambient Air Quality Monitoring
Program https://www3.epa.gov/ttn/amtic/qalist.html
3 Although the guidance focuses on 1-point QC checks since it is the only check currently reported to AQS. There
are other critical criteria that fall within the QA Handbook guidance
-------
• An invalid QC check is check in which there were technical issues with the generation of its test
concentration and the test concentration is not accepted as accurate.
A valid QC check which exceeds acceptance criteria (i.e., "fails") will result in at least some routine data
invalidation, but sometimes there is "compelling evidence" available regarding corrective actions and/or
additional analyzer checks that may not be readily viewable in the AQS dataset that helps bracket the
data set to be invalidated. We need to use, evaluate and report both valid and invalid QC checks in a
consistent manner.
The following two scenarios may exist for a monitor when a 1-point QC check has exceeded the
established acceptance criteria. A flowchart follows that describes these two scenarios:
Scenario 1
A 1-point QC check exceeds the established acceptance criteria. Upon investigation, the operator
determines that the 1-point QC check provided a valid concentration and that the analyzer needs
adjustment/calibration. This confirmation provides evidence that the 1-Point QC check was in fact a
valid check and, consequently, routine data should be invalidated.
Flagging Process for Scenario 1
1. The 1-Point QC check is reported to AQS, and the null code "EC" (exceeds critical criteria)
replaces the routine data either back to the last acceptable 1-point QC check or where additional
compelling evidence exits (see #2). "EC" is not the only null code that can be selected. Other null
codes more definitive of the monitors failure can also be selected (e.g., "AN" for machine
malfunction) in lieu of the "EC" null code. Where a monitoring organization responds to a QC
check exceedance with an adjustment/recalibration, it should be followed by another reported
QC check at the same concentration as the previous exceedance check in order for the routine
data between this valid passing check and the next scheduled check to be considered valid.
2. If there is compelling evidence (i.e., acceptable more frequent zeroes and spans which show a
clear pattern change, or other verification) to accept some of the data between the exceedance
and the last valid 1-point QC check, the routine data that was valid would be reported and
flagged "IV" (data was reviewed and validated). The compelling evidence needs to be
documented (described below).
3. EPA Regions, during the annual certification/concurrence process, will be able to evaluate the
information and flags used in this process. As mentioned below, with the development of a
quarterly report specifically related to these critical 1-point QC checks (and/or utilizing existing
AQS reports), EPA will be reviewing and evaluating these checks no less than quarterly.
NOTE: If routine data is invalidated using null codes, the valid QC check, although reported to AQS,
will not be used in aggregate statistics of precision and bias since the check is not representing the
precision and bias of the routine data for that time period.
NOTE: If no additional verification checks or other investigative measure to find compelling
evidence is performed on the analyzer or the QC system following the QC exceedance, then the 1-
point QC check will be considered valid and reported to AQS. EPA will consider the routine data
suspect and the data should be replaced with the "EC" null code back to the last passing check and
forward to the next passing check. Quarterly evaluation reports under development by EPA will
-------
highlight this data. If routine data is invalidated, the QC concentration, although reported to AQS,
will not be used in aggregate precision and bias calculations since the routine data representing the
exceedance was invalidated.
Scenario 2
A 1-point QC check exceeds the established acceptance criteria and there is compelling evidence to
consider the analyzer's data valid (and therefore consider the QC check invalid). For example, after an
acceptance criteria exceedance, the monitoring organization reviewed the data, went out to the site and
conducted an "as is" (no adjustment to analyzer) QC check, performance evaluation, or multi-point
verification at a concentration around the original QC check. These additional checks (not limited to the
examples described above) demonstrate that the analyzer is operating within the 1-point QC acceptance
limits and, therefore, supports the validity of the routine data. This compelling evidence also suggests
that corrective action is needed to the QC system that generated the invalid 1-point QC check. It is
suggested that corrective action be taken on the QC system immediately in order to determine the
definitive cause of the invalid check, which serves as further evidence to support the validity of the
routine data. A second acceptable 1-point QC check should be run so that routine data validity is
established from the acceptable check to the next scheduled 1-point QC check. It is also not expected
that a number of 14-day QC checks would exceed the acceptance criteria before corrective action is
taken to discover the there was a problem with the QC system and that the QC checks are invalid.
Flagging Process for Scenario 2
The following process is for gaseous pollutant data that exceed acceptance criteria of 1-point QC checks
(or Zero/Span) but monitoring organizations have compelling evidence to consider the routine data
valid.
1. The invalid 1-point QC check is not reported to AQS since the QC check is not considered valid.
Replace the QC check concentration with a "1C" null code. The code will create a "placeholder"
in AQS that will allow one to identify that a QC check occurred within the required 14-day
timeframe for data completeness purposes.
2. Document the evidence related to the invalid QC check. See compelling evidence described
below.
3. EPA Regions, during the annual certification/concurrence process, will be able to evaluate the
information and flags used in this process. As mentioned below, with the development of a
quarterly report specifically related to these critical 1-point QC checks (and/or utilizing existing
AQS reports), EPA will be reviewing and evaluating these checks no less than quarterly.
Compelling Evidence Documentation
Data flagged "IV" after a valid QC check exceedance or a null coded QC check ("1C") should be
documented in AQS. Two methods are available for this documentation:
1. Free form comments in AQS. This comment can be entered via the web application on the
maintain raw data form. EPA is not expecting a complete description of the issue and
resolution. Monitoring organization utilize instrument logbooks and site logbooks that should
provide the details of the flags used in this guidance. Free form notes can be a simple as
"Operator Error: mm/dd/yyyy" which can refer to a logbook entry date that could then be
discussed with the EPA Regions during data certification and reviewed during the next technical
systems audit. This is the preferred approach.
-------
2. Part of AMP600 certification and concurrence process. The AMP600 will be revised to identify
issues related to the QC exceedances and whether they have been handled as described in
scenarios 1 and 2. If not, the data will be flagged and require some compelling information.
Regions can then concur with the compelling evidence. Similar to #1 above, this information
can be a short description and refer to a log book entry.
Next Steps
Any routine data represented by failed 1-point QC checks that are not properly flagged in AQS will
be identified in EPA quarterly evaluation reports (currently in design phase). Attachment 1 is an
initial draft of what this quarterly report may look like. EPA Regions will work with monitoring
organizations on this data until a resolution of the validity of this data is reached prior to annual
certification/concurrence. Unresolved data issues represented by failed 1-point QC checks may not
be considered for regulatory use until completion of the annual certification/concurrence process.
In addition, 1-point QC checks will be evaluated for completeness in the quarterly reports to ensure
a check is performed and reported (via a concentration or a flag) every 14 days. It is strongly
suggested that these checks be automated and performed more frequently than every 14 days to
minimize loss of data due to invalidation. EPA Regions found monitoring organizations running
checks more frequently but not reporting them to AQS. We suggest all valid QC checks be reported
since it may also serve to minimize data invalidation.
EPA is in the review/development stages of this process. The "EC" null code and the "IV" qualifier
has already been developed by the National Air Data Group and can be used. They are working on
and the process to allow the reporting of the "1C" null code to the QA transaction by the end of the
year and we will alert monitoring organization in the event this is available sooner. The revision of
the AMP600 will be implemented for the 2018 data certification (may 2019) so will be in effect for
CY-2018 data.
-------
-------
Attachment 1
Development of Automated Quarterly Reports
DRAFT
Goal:
Provide an automated report that would allow OAQPS, Regions and monitoring organizations to
determine where there may be data quality issues in AQS that need to be reviewed evaluated and
potentially corrected on a quarterly basis.
Program Attributes:
The program will be developed in stages with the first stage devoted to those required QC checks
considered critical in the validation template of the QA Handbook and reported to AQS. Acceptance
criteria will be based on values listed in the 2017 Validation Templates.
Selection Criteria:
Data can be selected by a number of ways:
• Detail- Select "GEN" for a general table (Fig.l) or "DET" for a general table (Fig.l) and site by site
(Fig. 2) report
• Year/quarter- Can select a year, or a year and quarter
• Geographic area
o EPA Region- select "ALL" for all EPA Regions or by individual Region
o State- Can select state within an EPA Regions or leave blank
o PQAO- Can select PQAO within a state or leave blank
• Pollutant- either "ALL" for all criteria pollutants or select by parameter code
• Issues- Can either select "ALL" for all sites within the selection criteria or "ISS" to select those sites
where a failure has occurred and the data has not been appropriately flagged or null coded
• DV- Can either select "ALL" for all sites or "DV" to report for those sites at 80% and greater of the
design value
Output:
Figures 1 and 2 provide an idea of the reports generated based on the selection criteria. These figures
are for the ozone 1-point QC criteria.
Quarterly QC Report
Selection Detail:GEN; Year.2017;
gion: Region 1; PollutantiOzone; Issues ALlj DV;AII
Mlnumum QC
Number
EPA
Checks
QC
Number
QC
Routine Data
Site ID-POC
Site Name
Region
State
PQAO
DV 2014 2016
Required
Checks
QC Failed
Completeness
Completeness
QC Issues
500070007-01
PR0CT0H MAVLfc RfcStAH)
1
Vermont
61
7
9
2
100%
70%
NO
901S9991-01
Ahingifii-i
1
Connecticut
70
7
7
0
100%
100%
No
330099991-01
Woodstock
1
New Hampshire
54
7
7
0
100%
100%
NO
90019003-01
Sfwr**>od lilarid Coiw«
1
Connecticut
85
7
7
0
100%
100%
No
Z30039991-01
Ashland
1
Maine
52
7
5
0
71%
100%
No
230090102-01
TOP Or CADIUAC MTN
I
Maine
66
7
7
0
100%
100%
No
230130004-01
Marshall Point litfhthou
1
Maine
63
7
6
1
86%
85%
No
230310040-01
SBP Shaplctgh Ball Par
1
Maine
61
7
4
1
57%
85%
Yes
330012004-01
FIELD OFFICE ON THE GR
1
New Hampshire
58
7
7
0
100%
100%
No
330150018-01
MOO St Hill SCMOOl
1
New Hampshire
65
7
7
0
100%
100%
No
90010017-01
Greenwich Point Park
1
Connecticut
80
7
9
2
100%
78%
Yes
90011123-01
Western Conn State Uni>
1
Connecticut
78
7
7
0
100%
100%
No
Site had issues but "No"
means they addressed data
appropriately with correct
* flagging and invalidation (see
example)
>
In this case data is not
properly addressed.
See Fig 2.
Figure 1. Example of a Quarterly Report for 1-point QC Check (example only information fictitious)
-------
Report Details (Fig 1)
The report will highlight a few areas:
• Design Value Sites- Although you can sort and look at only sites from 80% and greater of the design
value if you select "ALL" sites in the DV selection, any sites in a general report that fall within the
criteria will also be highlighted (DV = All).
• QC Number Failed - Any site with a failed QC (whether the QC value was valid or not) will be
highlighted,
• QC Completeness- Any site with QC completeness <75% will be highlighted.
• Routine Data Completeness- Any site with routine data completeness <75% will be highlighted
• Issues - any site that did not provide correct flagging of their data or did not invalidate their data
(based on appropriate null codes and data flags) will be highlighted (see Fig. 2)
Site Report (Fig 2)
The site report will provide similar detail to the Region 3 504 Report in both a graphical (left) and a table
style. It provides some of the information in the general report but then provides more detail. Color
coding helps to provide information about the data reported to AQS and whether there is compelling
evidence to keep routine data that have failed a valid QC check. See the scenarios for the example site in
Figure 2.
2017 Quarterly 1-Point QC Report-Quarter 1
(Example only Information Fictitious)
PQAO Code- 0251 PQAO Name- Connecticut
Site ID 09-001-0017 Site Name- Greenwich Point Park
January
Su Mo Tu We Th Fr Sa
© Ifi I
115 if> 17 ® 20 2
22 23 24 25 26 27 2;
29 30 31
February
Su Mo Tu We Th Fr Sa
\7 13 14 15
19 20 21 22 23 24
26 27 28
Pollutant-Ozone
Design Value Site-Y QC Issue-Y
1-Point QC Checks Values
March
Su Mo Tu We Th Fr Sa
1 (D 3 4
5
6
7 8 9 10 11
12
13
14 15 @ 17 18
19
20
21 22 23 24 25
26
27
28 29
Passing QC Check Invalid QC Check (Flagged QC Value)
QC Check Exceedance (valid)
Date
% diff
Dates Effected
1/5/17
8.95%
12/18/16-1/18/17
1/19/17
3.5%
NA
2/2/17
6.2%
NA
2/16/17
7.8%
2/10/17-2/16/17
2/17/17
2.5%
NA
3/2/17
2.5%
NA
3/16/17
3.4%
NA
3/30/17
8.2%
NV
3/31/17
2.6%
NA
1-Point QC Completeness=100%
(due to extra QC samples on 2/17 & 3/31)
Invalid routine data NOT flagged "EC"
Invalid routine data flagged "EC"
Valid routine data flagged"lV"
Figure 2. Quarterly Site Report
Scenarios
1/5/2017- QC check exceedance (valid QC check) and monitoring organization properly invalidated data
and utilized "EC" null code
2/16/2017 - QC check exceedance (valid QC check) and monitoring organization found compelling
evidence to keep data from 2/3 to 2/9 and reported a "IV" qualifier on the routine data. However, they
left the remaining concentration data in AQS without a flag or null code. They recalibrated the analyzer
and ran another QC check the next day which passed.
-------
3/30/2017-A QC check failed but the monitoring organization went out and did an independent audit
and found the analyzer was within acceptance limits and found a leak in the calibration system.
Therefore, the QC Check was invalid and reported a "1C" null code to the QA transaction in lieu of a
measurement result. After fixing QC system they ran another 1-point QC check that passed.
------- |