oEPA
EPA/600/R-15/314 | November 2015 | www.epa.gov/ord
United States
Environmental Protection
Agency
Evaluation of Elm and
Speck Sensors
I Off ice of Research and Development
(National Exposure Research Laboratory
-------
EPA/600/R-15/314 | November 2015 | www.epa.gov/ord
Evaluation of Elm and Speck Sensors
Ron Williams
National Exposure Research Laboratory
Office of Research and Development
U.S. Environmental Protection Agency
Research Triangle Park, NC, USA 27711
Amanda Kaufman
ORISE Participant
Oak Ridge Institute for Science and Education
Oak Ridge, TN, USA 37831
Tim Hanley, Joann Rice
Office of Air Quality Planning & Standards
U.S. Environmental Protection Agency
Research Triangle Park, NC, USA 27711
Sam Garvey
CSS Dynamac
Research Triangle Park, NC 27711
-------
Disclaimer
This technical report presents the results of work performed by Alion Science and
Technology and Jacobs Technology under contracts EP-D-10-070 and EP-C-15-008, respectively,
for the Human Exposure and Atmospheric Sciences Division, U.S. Environmental Protection
Agency (U.S. EPA), Research Triangle Park, NC. It has been reviewed by the U.S. EPA and
approved for publication. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
in
-------
Acknowledgments
The National Exposure Research Laboratory's (NERL) Quality Assurance Manager (Sania
Tong-Argao) and associated staff (Monica Nees) are acknowledged for their contributions to the
development of standard operating procedures and the project's quality assurance project plan
(QAPP) used in execution of the research effort. This research was supported in part by an
appointment to the Research Participation Program for the U. S. Environmental Protection Agency,
Office of Research and Development (ORD), administered by the Oak Ridge Institute for Science
and Education through an interagency agreement between the U.S. Department of Energy and EPA
(DW 8992298301). Stacey Henkle, and Zora Drake-Richmond, (Alion Science and Technology)
are acknowledged for their contributions in supporting the field monitoring component of this
effort.
IV
-------
Table of Contents
List of Tables vi
List of Figures vi
Acronyms and Abbreviations vii
Executive Summary viii
1.0 Introduction 1
2.0 Materials and Methods 2
2.1 PM Field Evaluations 2
3.0 Field Evaluation Results and Discussion 4
3.1 PerkinElmer Elm 4
3.2 Carnegie Mellon Speck 10
3.3 General Discussion 14
4.0 Study Limitations 16
4.1 Resource Limitations 16
4.1.1 Intra-sensor Performance Characteristics 16
4.1.2 Test Conditions 16
4.1.3 Sensor Make and Models 16
v
-------
Tables
Table 1. Summary of PM Sensor Performance and Ease of Use Features 15
Figures
Figure 1. Placement of the Elm at AIRS 3
Figure 2. Placement of the Speck at AIRS 3
Figure 3. Trace of the Elm NO2 and CAPS NO2 sensors overtime 5
Figure 4. Five minute comparisons of Elm NO2 sensor and CAPS NO2 sensor 5
Figure 5. Elm NO2 Sensor vs the time of day 6
Figure 6. Trace of the Elm ozone sensor and the API T-265 ozone sensor overtime 7
Figure 7. Elm ozone sensor compared to the T-265 ozone sensor 7
Figure 8. Elm PMio compared to relative humidity 8
Figure 9. Elm PMio vs Grimm PIVh.s 9
Figure 10. Elm PMio vs time of day 9
Figure 11. Speck compared to relative humidity 10
Figure 12. Trace of the Speck and the Grimm overtime 11
Figure 13. Speck compared to temperature 11
Figure 14. Speck compared to temperature after a temperature correction has been applied 12
Figure 15. Speck vs Grimm without temperature corrections 13
Figure 16. Speck vs Grimm with temperature corrections 13
VI
-------
Acronyms and Abbreviations
AC/DC alternating current/direct current
ACE Air Climate & Energy
AIRS Ambient Air Innovation Research Site
°C degrees Celsius
CAPS Cavity Attenuated Phase Shift
FEM Federal Equivalent Method
FRM Federal Reference Method
hr hour
m meter
mb millibar
min minute
NAAQS national ambient air quality standards
NERL National Exposure Research Laboratory
NO2 nitrogen dioxide
63 ozone
OAQPS Office of Air Quality Planning and Standards
ORD Office of Research and Development
PM particulate matter
ppb parts per billion
QAPP quality assurance project plan
r2 coefficient of determination
RH relative humidity, i.e., water vapor content of air expressed as a percentage of vapor
pressure of water at a given temperature and pressure
RTF Research Triangle Park
s second
SD secure digital
SEVI subscriber identity module
VOC volatile organic compound
vn
-------
Executive Summary
Background
Particulate matter (PM) is a pollutant of high public interest regulated by national ambient
air quality standards (NAAQS) using Federal Reference Method (FRM) and Federal Equivalent
Method (FEM) instrumentation identified for environmental monitoring. The US EPA has been
evaluating emerging PM sensor technologies that might provide benefit to citizen scientists and
the scientific community-at-large. Such technologies are rapidly expanding, and new versions of
sensor devices previously examined by the US EPA are released by manufacturers. The results
described here represent an examination of two such examples involving the Creative Labs Speck
and the PerkinElmer Elm sensors.
Study Objectives
The US EPA's Air Climate & Energy (ACE) research program is engaged in an ongoing
effort to discover and evaluate a wide array of emerging technologies. In particular, it is conducting
world-wide market surveys of low cost PM sensors (<$2,500.00). Such a price point represents
the upper limit of cost that community groups and citizen scientists often see as the maximum
affordable expenditure regarding any capital investment they might make with respect to acquiring
low-cost sensors for their own use. The US EPA is conducting collocated field evaluations of select
sensors in direct comparison with FEM instrumentation. Selection is based upon the unique
features of the device that might provide technical insight into technologies not previously
examined, its commercialization and availability to the general public. Direct requests from US
EPA stakeholders (Regional offices, State air quality officials, etc.) desiring to gain knowledge on
specific sensors is also a factor in conducting this research. The devices examined in this report
reflect sensors previously examined in earlier efforts that have undergone significant revision by
the manufacturers. Likewise, the devices represented sensors either being widely used in citizen
science efforts or which were releasing data in a very public format. Therefore there was a high
degree of interest in their performance characteristics by a wide range of air quality officials. The
Creative Labs Speck and the PerkinElmer Elm sensors were obtained and sited in the established
PM sensor test platform on the US EPA's RTF, NC campus (AIRS). Data collections representing
approximately a 45 day evaluation period were conducted. The collocated PM2.5 FEM
instrumentation with 5-minute (min) time resolution provided the means to investigate both short
duration and daily (24-hour [hr]) comparisons between the test devices and the FEM response.
Potential data confounders such as temperature and relative humidity (RH) were obtained to aid
in the investigation. The relationship between FEM response and the various sensors was
established using regression formulas.
Vlll
-------
Study Approach
A single Speck PM sensor was obtained from Creative Labs and following review of its
updated software and data output characteristics (in comparison to the unit previously examined1),
quality assurance protocols were developed. Also, a copy of the PerkinElmer Elm pod was
obtained. Operating procedures were developed to acquire its data stream2. The Elm is a multi-
sensor pod device, capable of reporting on multiple air quality pollutants. The unit obtained for
this effort provided for ozone (63), nitrogen dioxide (NCh), and PMio estimates.
The necessary infrastructure to conduct the short-term evaluations was established. The
Elm is weatherized and capable of direct placement in the open environment (i.e., rain). It also
provides automated data transmission and processing via the manufacturer's software. The Speck
does not have these innate features. Therefore, weather shielding as well as data
collection/processing procedures were developed using previously established means for
deploying this device.1
For approximately one and a half months during the winter of 2015, these collocated low
cost sensors were sited on a monitor test platform with a Grimm Model EDM180 PIVh.s (EQPM-
0311-195) FEM on the US EPA's RTF, NC campus. Comparison research on FEM monitors,
capable of providing high degree confidence estimates of ozone or nitrogen dioxide, was also
accomplished at the test site. Both sensor and comparison monitors, along with ancillary
meteorological sensors (RH, temperature), operated continuously during this time. The only
exception was data recovery, flow checks/calibration, and general servicing as required by the
various manufacturers. Once the monitoring period was completed, data from the comparison
monitors and sensors was compared to determine how these variables influence low cost sensor
performance.
Sensor Performance Results
Discreet statistical evaluation of sensor performance (Speck and Elm) was established with
respect to collocated data associated with the Grimm FEM, as well as the comparison monitors for
nitrogen dioxide and ozone (Elm). Resulting regression characteristics were optimized with
respect to data normalization and influence of confounders under some circumstances. This was
an effort to account for observed sensor limitations with respect to environmental operating
conditions such as RH.
1 Williams. R.. A. Kaufman. T. Hanley. J. Rice. AND S. Garvey. Evaluation of Field-deployed Low Cost PM
Sensors. U.S. Environmental Protection Agency. Washington. DC. EPA/600/R-14/464 (NTIS PB 2015-1021041 2014.
2 Williams. R.. A. Kaufman. AND S. Garvey. PerkinElmer Elm. U.S. Environmental Protection Agency.
Washington. DC. EPA/600/R-15/125 (NTIS PB2015-105136V 2015.
ix
-------
Ease of Use Features Evaluation
Concerning ease of use features, several key findings were evident. In general, these
included, but were not limited to:
Power Requirements: Both of the units required basic electrical connections using step-down
transformed power, as they do not possess internal power sources.
Data collection/transmission/storage/recovery: The Elm has the ability to transmit data directly
to the manufacturer's website where it can be viewed online. We did not activate that option
with the device and instead collected data using the internal data storage card in the Elm. We
often find internal data storage cards provide significant benefit as compared to WiFi or cellular
options, and they do not hinder the comparison in any way. Speck data was collected via a laptop
using a direct cable connection between the two devices. Both the Elm and Speck raw data had
to be processed via manufacturer's software. This was accomplished via internet connections
where EPA collected data were transmitted to the manufacturer's proprietary analysis packages.
Processed data in the final format provided by the manufacturer were then transmitted back
automatically to EPA and used in the resulting analyses without modification.
Data Schemes: Data schemas (output) by the two manufacturers varied. Therefore, all processed
data were recovered and then integrated into an EPA-developed database to allow for
comparisons to be made between sensor data and reference monitoring data. All data was defined
by time/date stamps (1 minute integration periods) and represented the primary means by which
comparisons were established. Previous efforts concerning such data clearly indicated that longer
averaging times resulted in improved regression between reference data and sensor data, and 5
minute integration periods represented the primary means of comparison.
Installation and WiFi considerations: Sensors were not operated using wireless data
transmission either due to EPA decision, or the inability of the device (Speck) to operate in that
manner.
Conclusions
This marks the second formal evaluation we have performed on the Speck, and no
significant improvement in agreement with the reference monitor was observed (r2 < 0.1), when
fairly short time intervals (5 minute averages) are compared. RH in excess of 95% was shown to
have a dramatic impact upon PM concentration estimates. The device showed a marked and
significant response (positive bias) with respect to increasing temperature conditions. While no
statistical modeling was performed on one specific area of the data, it suggests that some of the
poor regression effects might be associated with the device providing poor agreement with the
reference monitor when lower ambient concentrations were encountered (PIVb.s < ~ 10 |ig/m3).
Model 2 of the Speck evaluated here had significantly improved output features in comparison to
the original model previously evaluated, relative to its onboard display panel. Likewise, Creative
Labs output processing software associated with the current model offered some advantages with
respect to features end users might appreciate.
-------
We previously evaluated the CanairIT sensor pod.1'3 The Elm appears to have many of the
same physical features as the CanairIT device. While the original Canairlt and Elm had some
similarities (size, appearance, etc.), differences related to proprietary configurations between the
two pods cannot be fully defined here, and are not theorized. In particular, the sensor elements
themselves might have changed as well as any algorithm or response factors used in reporting air
quality outputs.
Poor agreement (r2 < 0.01) was established here between the Elm and NO2 monitor
comparison. The unit's ozone sensor provided the most agreeable comparisons with our reference
measurements (r2 > 0.7) as compared to any sensor making up this pod. There was no general
agreement between the Elm and its PM measures with respect to the collocated reference monitor,
relative to any discernable pattern. RH events in excess of ~ 90% were shown to influence the
response.
3 Williams. R.. A. Kaufman. AND S. Garvey. Next Generation Air Monitoring (NGAM) VOC Sensor
Evaluation Report. U.S. Environmental Protection Agency. Washington. DC. EPA/600/R-15/122 (NTIS PB20I5-
1051331 2015.
xi
-------
1.0 Introduction
EPA's Office of Research and Development (ORD) has been engaged in the discovery of
low cost sensors potentially useful for air quality monitoring.4 As defined in a sensors users guide
focused on potential end users, the performance characteristics of many of these devices entering
the market or public domain have not been reported.5 To assist citizen scientists, state, municipal,
federal air quality officials, as well as sensor developers, the US EPA has been evaluating select
sensor devices in a series of laboratory and/or field monitoring research studies. To date, these
include sensors associated with PM, ozone, nitrogen dioxide, and volatile organic compounds.1'3'6
The US EPA has focused a majority of its attention on sensors costing < $2,500.00 as it is
believed such a cost would be at the upper limit to that which citizen scientists, as well as many
others might be able to afford. Even so, it must be recognized that efforts such as those reported
above represent a limited survey of all the sensor technologies currently being manufactured. One
feature of this market is that rapid advances are being seen in product development. It is not
unusual for devices to be released in multiple versions within the same calendar year. These
include revisions to the sensor's physical features (such as changes in the base sensing element(s)
itself or the data processing algorithm). Therefore attempting to stay current with the performance
characteristics of any one sensor is almost an impossibility due to time and resource limitations.
The work being reported in this study, represents our efforts to revisit two sensors (or
earlier versions of these sensors) that are being employed in communities or citizen science
activities. Both of the sensors had undergone significant changes and interest from multiple EPA
stakeholders encouraged their re-examination.
4 MacDonnell. M.. M. Raymond. D. Wyker. M. Finster. Y. Chang. T. Raymond. B. Temple. M. Scofield. D.
Vallano. E. Snyder. AND R. Williams. Mobile Sensors and Applications for Air Pollutants. U.S. Environmental
Protection Agency. Washington. DC. EPA/600/R-14/051 (NTIS PB20I4 105955V 2014.
5 Williams. R.. Vasu Kilaru. E. Snyder. A. Kaufman. T. Dye. A. Rutter. A. Russell. AND H. Hafner. Air Sensor
Guidebook. U.S. Environmental Protection Agency. Washington. DC. EPA/600/R-141159 (NTIS PB20I5-I006IO).
2014.
6 Williams. R.. R. Long. M. Beaver. A. Kaufman. F. Zeiger. M. Heimbinder. I. Hang. R. Yap. B. Acharya. B.
Ginwald. K. Kupcho. S. Robinson. O. Zaouak. B. Aubert. M. Hannigan. R. Piedrahita. N. Masson. B. Moran. M. Rook.
P. Heppner. C. Cogar. N. Nikzad. AND W. Griswold. Sensor Evaluation Report. U.S. Environmental Protection
Agency. Washington. DC. EPA/600/R-14/143 (NTIS PB20I5-I006I I). 2014.
1
-------
2.0 Materials and Methods
Two sensors were obtained by direct purchase for PM field evaluations. This included the
Carnegie Mellon Speck, which had recently received a major update, and the PerkinElmer Elm,
formerly distributed in a version known as the Airbase CanarlT. The latter had been updated to
some unknown degree since the company's acquisition and market distribution through
PerkinElmer. The NO2 and Cb measurement capabilities of the Elm were also measured in this
field evaluation. The evaluation sought to compare these sensors against Federal Reference and/or
Federal Equivalent Methods (FRM/FEM). The effects of temperature (° C), relative humidity (%
RH), wind speed (m/s), and pressure (mb) were explored as possible interferences.
The coefficient of determination (r2) is the square of the sample correlation coefficient and
was used as a measure of linearity. Microsoft Excel was used to calculate r2 by plotting all
measured values against data acquired by FRM/FEM, which displayed the linear regression. The
same plot was used to determine response factors and offsets.
2.1 PM Field Evaluations
A Grimm Technologies, Inc. (Douglasville, GA) Federal Equivalent Method (FEM)
analyzer was operated by EPA's Office of Air Quality Planning and Standards (OAQPS) alongside
meteorological instrumentation, an API T-265 63 analyzer, and a cavity attenuated phase shift
(CAPS) NO2 analyzer at the AIRS monitoring station on the EPA campus in Research Triangle
Park (RTF), NC. Specifics about the description and basic operation of the model T500U CAPS
NO2 analyzer (Automated Equivalent Method: EQNA-0514-212) is described elsewhere.7 These
established reference methods are covered under a QAPP for that study (EPA, 2013).8 Reference
data were available for the time frame of the sensor evaluation as 5-min averages.
The Elm was attached to a pole mounted to the AIRS platform railing as shown in Figure
1. Zip ties were used in conjunction with the mounting bracket supplied by the manufacturer to
attach the unit to the pole. The manufacturer-supplied rain shield was deemed sufficient to protect
the Elm from the elements. There was not an active data contract for the Elm's subscriber identity
module (SIM) card at the time of testing. Data were recovered from the micro secure digital (SD)
card located inside the unit and sent to the manufacturer for processing. The manufacturer
responded with the processed data in 5-min averages. The Elm ran without interruption from
2/13/2015 to 3/30/2015 with 1-min data being collected.
The Speck was placed inside one of the Bowl-on-Pole shelters described in an earlier report
as shown in Figure 2.1 The unit was placed on the shelter grating such that one of the large holes
in the grating was directly beneath the bottom mounted air intake of the sensor. The micro-USB
cable which supplied both a power and data connection for the unit was connected to a laptop
7 Federal Register: Vol.79, pages 34734-34735, 06/18/2014
8 U.S. Environmental Protection Agency (EPA). July 2013. QAPP. Raleigh Multi-Pollutant Near-Road Site:
Measuring the Impact of Local Traffic on Air Quality. Research Triangle Park, NC.
-------
computer running the Speck Gateway software (April 2015 version). The laptop was located in a
weather protected shelter. The Speck ran without interruption from 2/13/2015 to 3/30/2015 and
provided 1-min data averages.
Figure 1. Placement of the Elm at the AIRS
. M^.
Figure 2. Placement of the Speck at the AIRS
-------
3.0 Field Evaluation Results and Discussion
3.1 PerkinElmer Elm
The Elm has three on board sensors which were evaluated in this study: NO2, Cb, and PMio.
The true PM size designation of the Elm is not fully known. The output software had column
headers for both PIVh.s and PMio, but only data associated with the PMio designation was received
following processing. We therefore report it here as PMio, with the caveat above as to its particle
size uncertainty. We compared the Elm output to PM2.5 reference measurements based upon the
availability of such measurements for this research effort.
The NO2 sensor output over time is shown in Figure 3 superimposed with that of the CAPS
reference NO2 sensor. The two traces revealed minimal overlap over time, even notwithstanding
the fact that their y-axes are on scales differing by a factor of 10. In some instances, the Elm
reported 5-min integrated NO2 values in excess of 3000-4000 parts per billion (ppb). Usually these
response upsets were short lived (on the order of 5-10 minutes in total duration) before a more
normal (~ 50 ppb) response was evident in the data pattern. The reference monitor never revealed
any similar pattern of response. These occasional periods of very high response values impacted
the resulting regression and the figures presented herein reflect these observations without
censoring.
The reference monitor reported significantly lower values as compared to the Elm. Figure
4 compares the CAPS reference NO2 analyzer to the Elm's NO2 sensor directly, and revealed no
significant correlation between the two monitors. Possible relationships between the Elm's NO2
sensor and wind speed, temperature, RH, and pressure were explored. No significant correlation
between the sensor's response and those meteorological parameters were established.
While the Elm's NO2 sensor response could not be correlated to the reference measurement
or meteorological parameters, we investigated the output to determine if any pattern whatsoever
existed. Doing this might provide insight as to the sensor's response relative to it being the result
of some undefined cofactor or simply noise. To simplify this investigation, the data were broken
up into blocks based on the hour of the day. Each of the resulting 24 (1-hr) blocks were averaged
and plotted in Figure 5.
-------
(Q
c
(D
3
5'
&
(D
O
O
o
03
2_
55'
o
(A
s,
m
3
(A
(D
3
(A
O
^
D)
Q.
O
>
TJ
CO
(A
(D
(A
O
Elm (ppb)
NJ
Ln
Ln
88888888
o
Kl
c/i
n>
oo
O
^
to
n
o
3
o>
ro
Q.
CAPS NO2 Concentration (ppb)
h-> h-> hJ hJ U) U)
OLnOLnOLnOLn
(Q
c
3
D)
O
(D
(D
m
3
z
O
ro
D)
Q.
O
TJ
CO
(A
(D
3
(A
O
3)
§
(D
3
(D
Elm NO2 Concentration (ppb)
-------
1.6
1.4
1.2
1 1
fo.8|
1 0.6
LLJ
0.4
0.2
0
(
Elm N02 vs Time of Day
4
I
t
(
)
<
,
)
) 5 10 15 20
Time of Day (Hour)
Figure 5. Elm NC>2 Sensor vs the time of day
It would appear that some base response mimicking diurnal patterns of ambient NO2
patterns did exist with the Elm output. Such an output would normally be expected to occur. Even
so, as evident in Figure 5, the overall response was low and at a level where the limit of detection
was not being routinely achieved.
The Elm's onboard ozone sensor response over time is shown in Figure 6, superimposed
with the API T-265 ozone analyzer (used as a reference). The two appear to show strong agreement
with the T-265 generally reading slightly higher than the Elm. This is confirmed in Figure 7, which
compares the two sensors directly. A coefficient of determination of 0.73 confirms that a strong,
fairly linear correlation existed between the two. The response factor of 0.875 and an offset of 6.2
both support the observation that the Elm underrepresents the ozone levels compared to the T-265.
Correlations between the Elm ozone sensor and meteorological data were explored. A negative
correlation was found with relative humidity. However, on further examination a nearly identical
correlation was found between the T-265 and relative humidity. This suggests that the correlation
was a real phenomenon and not a bias or error inherent to the Elm's onboard ozone sensor.
-------
Ozone Sensors Over Time
ELM T-265
2/13 2/18 2/23 2/28 3/5 3/10 3/15 3/20 3/25 3/30
Figure 6. Trace of the Elm ozone sensor and the API T-265 ozone sensor over time
Ozone Sensors Compared
y = 0.875x-6.222
R2 = 0.73
T-265 Reference (ppb)
Figure 7. Elm ozone sensor compared to the T-265 ozone sensor
The Elm is equipped with an optical PM sensor. Optical PM sensors are known to
frequently suffer from meteorological interferences. Most notably, high relative humidity is known
to produce artificially high values in such sensors. For this reason, correlations with meteorological
7
-------
conditions were first explored. No significant correlations were found with wind speed,
temperature or pressure. Relative humidity measurements of greater than 95% were found to be
correlated with a spike in response as shown in Figure 8.
700
600
^ 500
rft
}g 400
S
| 300
E
m 200
100
0
1
Elm PM10 vs Relative Humidity
MM*
5 2
y = 0.
R
014x + 0.0
2 = 0.002
06
^^_ ^^^ ^^^ ^^^
5 35 45 55 65 7
Relative Humidity (%)
(
5 8
L_A_
5 95 1C
)5
Figure 8. Elm PM compared to relative humidity
The Grimm is not the ideal reference for the Elm, because this particular model of the Elm
only provides PMio data while the Grimm provides PM2.5 data. Still, it was expected the two would
show some general agreement as the PIVb.s size fraction is typically the dominant particle size
associated with the test location. Figure 9 reports no general agreement between the two devices.
Even if the Elm's PMio sensor did not correlate with the reference measurements, observation of
some recognizable response over time would support the idea that it is measuring real phenomena
instead of just noise. To investigate this potential, the data were broken up into blocks based on
the hour of the day. Each of these 24 (1-hr) blocks were averaged and plotted in Figure 10. The
distribution appears to reflect some aspects of a diurnal pattern that one might relate to general
photochemical PM development over the course of a given day.
-------
60
50
40
o 30
Q.
£ 20
10
Paniculate Sensors Compared
y =
0.005X +0.516
R2 = 0.0001
10 15
Grimm PM25 (ng/m3)
Figure 9. Elm PM vs Grimm PM2.5
40
35
30
m
-125,
|
S20
I15
LLJ
10
5
0
(
Elm PM10 vs Time of Day
>
t
9
<
I
1
w
1
..
.
I
) 5 10 15 20
Time of Day (Hour)
Figure 10. Elm PM vs time of day
-------
3.2 Carnegie Mellon Speck
The Speck is an optical particle counter with an onboard algorithm to convert particle
counts to PM2.5 concentration. Relative humidity is a known interferant with optical particle
counters, so its effects were first explored in Figure 11. Data collected above 92% relative humidity
were found to have extremely high values far exceeding the true ambient concentration. Thus, all
data taken when the relative humidity was above 92% were removed from the analysis.
Relative Humidity vs Concentration
160
140
20
30
40 50 60 70
Relative Humidity (%)
80
90
100
Figure 11. Speck compared to relative humidity
The concentration over time is shown in Figure 12, superimposed with that of the Grimm
reference sampler. The two traces over time sometimes appear similar but diverge other times.
One of the primary points of divergence is that the Speck has frequent clusters of data at very low
values. These clusters are not seen in the Grimm data. These clusters do not seem to correlate with
any meteorological condition. Attempts were made to remove all Speck data below a threshold
value. Threshold values of both 0.1 ng/m3 and 1 ng/m3 were attempted, but neither attempt was
found to improve the correlation between the Speck and the Grimm. Therefore, these attempts
were not reported here.
10
-------
PM2 5 Sensors Over Time
25
s
20
§ 15
2 10
2/13 2/18 2/23 2/28 3/5 3/10 3/15 3/20 3/25 3/30
Date
Figure 12. Trace of the Speck and the Grimm overtime
Meteorological conditions were also explored for correlations with the Speck data as a
whole. Temperature was found to have significant correlation with the Speck as shown in Figure
13. The Grimm was also checked for a correlation with temperature, but none was found.
Temperature vs Concentration
21
Temperature!
Figure 13. Speck compared to temperature
11
-------
A correction to the Speck data response was made based on the temperature correlation.
First, the average of all Speck data was measured to be 3.13 jig/m3. Then a constant, 1.43 |ig/m3,
was added to the Speck data such that the Y-intercept of the temperature correlation best fit line
would match the average of all Speck data. That best fit line was then subtracted from the Speck
data. The result was that the corrected data plotted against temperature would have no temperature
dependence and would have an average equal to the average of all of the original Speck data as
shown in Figure 14.
Temperature vs Corrected Concentration
Temperature!
Figure 14. Speck compared to temperature after a temperature correction has been applied
This correction did improve the correlation between the Speck and the Grimm, but this
correlation remained poor as shown in Figures 15 and 16.
12
-------
Grimm vs Speck
25
20
15
10
(J
01
Q.
Ifl
y = 0.074x + 2.690
R2 = 0.006
10 15
Grimm PM2 5 (|ig/m3)
Figure 15. Speck vs Grimm without temperature corrections
Grimm vs Corrected Speck
20
y = 0.146x +2.276
R2 = 0.039
Grimm PM2 5 (|ig/m3)
Figure 16. Speck vs Grimm with temperature corrections
Even with these corrections applied, the strongest coefficient of determination with the
Grimm was only 0.04. The Speck does appear to visually track the Grimm at times, but cannot be
shown to do so quantitatively. There remains a great deal of seemingly random error overlaid on
13
-------
top of the signal. One prominent source of error appears to be in the form of clusters of extremely
low values. If this source of error can be eliminated, it is likely that correlations might improve
markedly. A temperature correction factor built into the software would also likely be of value.
Finally, it must again be noted that a major update to the Speck has been produced since these data
were collected. This latest version of the Speck is currently deployed in an ongoing Denver sensor
evaluation (CAIRSENSE). Data from the CAIRSENSE will not be available until well into 2016
and therefore no distinction between the version of the Speck evaluated here and the most current
version can be provided.
3.3 General Discussion
The general performance features of the two sensors evaluated here is summarized in
Table 1. The terms used in the table are defined as follows:
RH limit: the highest relative humidity at which the sensor can produce reliable data.
Temp Effects: if a direct relationship exists between temperature and the sensor's
signal, the R2 of that relationship is displayed.
Time Resolution: the measure of how frequently the sensor produces a data point.
Uptime: qualitative assessment by the operator about the frequency of data loss.
Ease of Installation: qualitative assessment by the operator about the level of effort
required to bring the sensor to operational status in the field.
Ease of Operation: qualitative assessment by the operator about the level of effort
required to operate the sensor, take data, and process the data.
Mobility: qualitative assessment by the operator about the level of infrastructure
required to operate the sensor in the field using the current research operating
procedure. Other procedures might have different requirements.
It should be recognized that uptime, ease of installation, ease of operation, and mobility
descriptors provided here are somewhat arbitrary as no definitive criteria exist for their
quantitation. As reported here, they define what we observed when trained technical staff
attempted to operate the device in an outdoor environment. As an example, uptime rating was
highly dependent upon the ability of the device to maintain data collection operations for an
extended period of time. An excellent rating would indicate near flawless data collection
capability. Ease of installation was influenced by how quickly the device could be placed
outdoors as provided directly from the manufacturer. A poor rating is indicative of the need to
work well beyond the primary directions provided by the manufacturer to establish basic data
collection operations. Ease of operation was defined as how easy it was to start, complete and
recover data collections. A fair rating was indicative of the fact that such operations were
eventually completed but with some effort needed to make this a repetitive process. Lastly,
mobility was defined as how easy it would be to move the device from one location to another. A
poor rating would equate to a sensor that had to be hard wired to a computer, an alternating
14
-------
current/direct current (AC/DC) power supply, or other features (e.g., weather shielding, WiFi
hotspot) that would limit the ease of movement with respect to successful data collections.
Table 1. Summary of PM Sensor Performance and Ease of Use Features
Sensor
PerkinElmer
Elm
Carnegie
Mellon
Speck
(particle
counts)
RH Limit
Impacts
observed at
RH > 95% for
PM
measuremen
t
Impacts
observed at
RH > 95%
Major
temp
effects
Not
observed
Observed
Time
resolution
1 min
1 min
Uptime
Excellent
Good
Ease of
installation
Good
Fair (rain
shielding)
Ease of
operation
Excellent
Good
Mobility
Fair
Fair
In general, the Elm was easy to operate once it was positioned at the collocation monitoring
platform. No general maintenance or servicing was required once it began operation with the
exception of data downloads. It did require access to a land-based power source. In its normal
operation, the unit requires access to cellular data transmission. We harvested data directly from
the unit without cellular service and therefore cannot share any findings relative to communication
uptime and transmission activities. Data processing occurred through a file being returned to the
manufacturer and the subsequent concentration values/time stamps being returned to us.2
We had some issues with units of the Speck operating successfully prior to initiation of this
effort but once an operational unit was obtained it collected data without interruption throughout
the study period. We had to protect the unit from weather events as interaction with direct
precipitation would result in instrument failure. Processing of the data was performed via access
to the manufacturer's software for such purposes.
15
-------
4.0 Study Limitations
It must be recognized that the scope of this sensor performance evaluation was limited with
respect to a number of primary parameters:
The resources of the US EPA to conduct the extensive field tests defined herein, and
The scope of the performance testing was not meant to fully compare the devices versus FEM
standards.
4.1 Resource Limitations
4.1.1 Intra-sensor Performance Characteristics
This effort was not intended to be a definitive evaluation of the two devices. In particular,
only single units of each device were evaluated and therefore the potential for poor performance
to uncharacteristically reflect the sensor in general could exist. Therefore, this report provides very
limited findings on intra-sensor performance characteristics. As with any examination of data
precision, a sufficient amount of information from multiple instruments is necessary to truly assess
the ability of a monitoring device to accurately measure the challenge concentration and to do so
in a repeatable manner. Likewise, it has been our experience that low cost sensors sometimes fail
without any obvious warning and therefore the findings being reported here may reflect
comparisons not truly representative of the device's normal performance characteristics. We can
only assume that the devices operating here were functioning properly based upon their normal
operating guidelines and lack of fault indicators (if such warnings were available). We operated
units in a laboratory setting for 3-5 days prior to their field placement to ensure basic operational
status conditions were evident (data being transmitted or internally stored) and to provide for staff
familiarization of the device. Once units were placed in the field, we inspected them on a weekly
basis for operational status. We observed no obvious failures during field deployment.
4.1.2 Test Conditions
Resources prevented the US EPA from examining the sensors under a wide variety of
environmental and interfering agent conditions. Field evaluation was performed only during cold
weather (winter) seasonal conditions. This limited variability of temperature and relative humidity
conditions certainly restricts the extent results here might be extrapolated to represent other
seasons, namely summertime.
4.1.3 Sensor Make and Models
We have been made aware of an updated version of the Speck that was released after the
version evaluated in the current report. Based upon conversation with the developers, it is believed
this new model has an improved response algorithm. This latest version of the Speck is currently
being operated in a multi-seasonal evaluation as part of an US EPA sensor evaluation study
(CAIRSENSE) in Denver, CO. The CAIRSENSE will not yield preliminary data findings until
mid- to late summer of 2016 and therefore cannot provide any benefit here pertaining to Speck
performance. The Elm, based upon what is given at the manufacturer's website
(www.Elm.perkimElmer.com/), at the time of this report (November 2015), has been deployed in
16
-------
multiple world-wide locations. No specific information is readily available at this website
concerning the technology associated with the Elm, the types of sensors associated with the unit
evaluated here, or how it might have compared with earlier versions of the system.
17
-------
SB*
United States
Environmental Protection
Agency
Office of Research and Development (81D1 R)
Washington, DC 20460
'Official Business
Penalty for Private Use
$3430|
PRESORTED STANDARD
POSTAGE & FEES PAID
EPA
PERMIT NO. G-35
------- |