KK?
Why 2K? Because sometimes one K is just not enough.
- Kris Kringle
ICR Update
Jim Walasek, Editor
Technical Support Center
December 31, 1999
Special Millennium Issue
ICR Update ISSUe Number 20 -This information sheet, the ICR Update,
is the twentieth one to be issued by the Technical Support Center (TSC) of the Office of Ground
Water and Drinking Water (OGWDW). Future issues (if any) will be distributed as needed to
maintain information flow related to the ICR.
Editor's Note: Recently, when I've spoken to ICR Technical Contacts at water utilities affected
by the ICR they almost always express relief that the ICR is finally "wrapping up" after all these
years. Remember the rule was proposed in February of 1994 and then the final ICR rule was
promulgated on May 14,1996. Most of the utilities received their "applicability" letters back in
June of 1996. Sampling began in July of 1997 and here we are about to enter the new
millennium (I know, it's not really the new millennium until 2001, but the big parties will
happen as we enter 2000) with 18 months of ICR sampling completed and the final reports on the
way. In fact, eighteen months of ICR data have now been validated and verified and final
reports have been sent to ICR utilities and labs. The ICR data is being used to support regulatory
impact analysis and to validate the water treatment plant model.
A large percentage of the TSC staff has worked on this project full time since the git-go. My
involvement with the ICR actually goes back to 1994 when I worked on the development of the
sampling video for protozoa and viruses plus the companion guide. In 1995 TSC developed the
virus monitoring protocol video as well as the ICR Protozoan Method video (both of these also
had companion guides). About this time we also began work on the videos that introduced the
ICR Water Utility Database System and the ICR Laboratory QC Database System which had
been under development since 1994. TSC also developed several manuals to assist the ICR
community in the implementation of the ICR (ICR Sampling Manual^ ICR Microbial Laboratory
Manual, DBP/ICR Analytical Methods Manual, ICR Manual for Bench-and Pilot-Scale
Treatment Studies, and the ICR Reference Manual - Understanding the ICR). The TSC
-------
chemistry team developed and implemented the ICR laboratory "approval" procedure (including
the PE program) and provided technical assistance throughout the ICR. The same is true for the
microbial coordinator. The treatment studies coordinator developed the manual, reviewed
treatment study plans, provided technical assistance, developed the reporting software, reviewed
final reports and even designed the database to analyze the data. Finally, our computer guru
played a big role in the development of the ICR Federal database (ICRFED), developed the
criteria and applicability determinations for the ICR, and ensured that the ICRFED performed as
originally envisioned. So, as you can see, our involvement in the ICR here at TSC goes back
many years, and, YES, we too are VERY happy to be wrapping things up.
The last ICR Utility Data Validation Packages for Sampling Periods 16,17, and 18 (October,
November, and December 1998) were sent out for review in November with the last set of
resubmission diskettes due back just before Pearl Harbor Day. Final Reports for the last
quarter of ICR data were scheduled to be sent to utilities and labs between Christmas and New
Year's Day. So even with the slow down in mail delivery over the holidays they should arrive
early in the new millennium assuming, of course, that there is nothing to this Y2K scare.
Letters were sent out in early December to approximately 100 utilities that had not yet sent
in their Final Design (SP19) diskettes. Remember, these diskettes should have been sent in with
the last of the ICR monthly data transfer diskettes (end of April '99). We had requested that
they submit the Final Design diskettes by the middle of December so that data could be included
in the 18 month version of the AUX1 database. If you haven't sent in your Final Design diskette
yet, contact me ASAP at 513-569-7919. Thanks.
Well, we have finally reached the end of the ICR data gathering effort. Over the next few
months we will be continuing our data correction effort by revising and testing the validation,
posting, and reporting software. We will also begin to take an in-depth look at the QC data,
paying particular attention to precision and accuracy. Finally, in early April, any straggler
diskettes remaining will be uploaded and validation will be run for the final time.
I want to personally thank all of you who worked so hard to make the ICR a success. The
ICR data probably represents the highest quality data ever collected for an information gathering
effort such as this. Over the 18 months of sampling, some 1.5 million analytical results were
produced and submitted on over 15,000 diskettes (and those are just the ones that didn't get lost
in the mail). My guess is that the hard copy reports sent out to the utilities and labs, if stacked
vertically would exceed 700 feet. (I don't even want to think of the number of trees sacrificed.)
Thanks again, we couldn't have done it without you!
So ... What Happened to My Treatment Study Report
Anyway? - Good question — I'm glad you asked! After we received the Final Treatment
Study Reports in July 1999, or shortly thereafter in most cases, we conducted a thorough review
of the data. Anyone that conducted a treatment study has heard from us by now and responded
to our request to supply missing data or to verify questionable results. The review process has
taken longer than anticipated but has resulted in the correction of much erroneous data. At this
-------
point we are wrapping up the final reviews and hope to have them complete by the end of
1999.
While completing the review phase of this project, we have transitioned into the analysis
phase of this project. During analysis of GAC study results, we are using a logistic curve form to
describe the "S" shaped breakthrough pattern observed for most DBF precursors. During
analysis of membrane study results, we are evaluating both the permeate water quality as well as
the productivity of the membrane process. The results of these analyses will be managed in a
relational database that will be used to support the Stage 2 D/DBP rule development process.
Data analysis is proceeding smoothly, and we hope to complete this phase of the project in early
2000.
When we have completed our analysis, we will generate a distribution version of the ICR
Treatment Study Database which will be made available to the drinking water industry. The
information in this database on GAC and membrane performance will have tremendous utility
far outside the context of the regulatory development, and it is your hard work that made it
possible.
As we come to the close of this project, I want to extend my thanks and appreciation to
everyone involved in this effort. It has been a loooooong four years, but in that four years I have
enjoyed interacting with you as we worked through these studies. If you have any questions
regarding the treatment studies, GAC, membranes, disc golf or cycling (both road and mountain)
give me a call (513-569-7131). Ask for Steve.
We get letters ! - A couple of days ago Mary Ann Feige and I (the ICR Update
editor) received an e-mail letter from Jennifer Clancy of Clancy Environmental Consultants
regarding the protozoan PE results. I have printed Jennifer's letter and Mary Ann Feige's
response in their entirety.
Dear Jim and Mary Ann:
I have been meaning to write to you to express my concerns about the manner in which the EPA protozoan
performance evaluation (PE) results are being analyzed and communicated. I was concerned about the use of
the PE data when we received the September 1999 report from Mary Ann and am even more concerned to see
how the data were portrayed in the ICR Update from TSC, which treats the PE program as a de facto method
comparison study.
The coversheet sent by Mary Ann with each round of results states that "It is not a laboratory evaluation
program" and "that there are no pass/fail criteria." This is true. As I recall, the purpose of the EPA PE program
was to provide independent PE samples for laboratories to use to develop and maintain their own QC database.
My understanding was that laboratory-specific results without the data summaries and method
comparisons were to be sent to individual laboratories to enable them to assess their performance and identify
potential performance issues.
Instead, the PE results from a wide range of laboratories, many of whom began analyzing protozoa samples only
in the last several months, are being summarized and analyzed to draw conclusions regarding the relative
performance of Method 1623 and the ICR Method. The data collected through the PE program are unsuitable
for use in a valid method comparison study for the following reasons:
-------
1. There is no control over how the PE samples are used. We have found that it is critical to analyze these
samples upon receipt as storage affects the quality of the (oo)cysts.
2. Laboratories may or may not be following the method exactly as described. There is a great tendency in
protozoan analyses to improvise and tweak the method. Since these data are not used for compliance and
hence do not require strict adherence to the method, it is likely that laboratories do take liberties with the
method.
3. Any laboratory, regardless of skill level, can participate. Many of the laboratories have never been part
of a protozoan approval process including on-site visits so the quality of their operation is in question.
Remember that several of the laboratories with the longest experience in protozoan testing failed the ICR
approval process TWICE. Even if laboratories submit IPR results to confirm that they can perform the
method acceptably to enter the program, there is no way to confirm that the IPR data are legitimate.
4. The ability to properly identify (oo)cysts is critical, yet it is apparent from the false positive results that
those laboratories either have contaminated their samples or cannot properly identify the protozoa from
the other confounding objects which occur commonly.
5. Laboratories analyzing samples using the ICR method tend to be those with extensive experience in
protozoa analyses before and during the ICR, while most of the laboratories who are just now beginning
to perform protozoa analyses are using Method 1623. There is a learning curve with Method 1623, even
if one has previous experience with the ICR Method.
To do a proper method comparison study, laboratories would be pre-qualified as they were to participate in the
1622/1623 validation studies by prior on-site evaluations and a qualifying PE data round. This would include
initial and ongoing on-site audits to confirm laboratory performance and expertise, with particular attention to
microscopy skills. It is understandable why EPA is not undertaking this exercise. However, a valid method
comparison study would isolate the variable of interest the relative performance of each method from other
confounding variables, such as vastly different laboratory experience, laboratory-specific modifications and
tweaks to the protocol, different spike doses, etc.
By providing the summary data and comparing the two methods, EPA has made this into a methods comparison
study. The title of Jim's piece in the October 1999 ICR Update is "The Jury's Still Out" and goes on to describe
that the two methods show no significant difference. If this were truly the case, it makes me wonder just why
we needed Methods 1622/1623. These reported ICR Method recoveries are quite different than those seen in
the ICR under the EPA's own approval program. Although the use of the PE program data for method
comparisons may be an effort to provide all the data possible to laboratories to review, this is an improper use
of the data generated from these PE samples.
I would ask that EPA seriously reconsider the current approach of summarizing individual and largely
incomparable laboratory PE sample results and packaging them as method comparison data. The data generated
through the PE program do not constitute a method comparison study, and should not, under any circumstances,
be construed this way.
Despite these serious concerns, I laud EPA's efforts and initiative in implementing the protozoa PE program,
which fills a very real void for laboratory performance verification, now that the ICR has ended, and laboratory
certification is not available.
Sincerely,
Jennifer L. Clancy, Ph.D.
President
CLANCY ENVIRONMENTAL CONSULTANTS, INC.
-------
Mary Ann Feige's response:
Dear Jennifer,
Thank you for your letter. I understand your concerns and would like to address each one.
1. In the PE study, laboratories are required to provide bench sheets along with their results. All holding
times in the methods must be met for the analysis of PE samples; If holding times are not met, the data
are not used in the overall summaries.
2. If laboratories are not following the method exactly, we have asked them to send us their deviations or
if they are using a method other than 1622/1623 or the ICR method, we have asked them to send us a
copy of their method. It is true that laboratories may not be following the methods exactly, but there
is no guarantee that this would be the case under any program. Even in the ICR, we identified some
deviations which were discovered during the follow-up audits.
3. Although we do require laboratories to provide us their initial precision and recovery data (IPR), it is
true that many of the laboratories are relatively inexperienced, and this was noted in our summary and
our article in the last update. The only way to confirm that the IPR are legitimate is to observe the
laboratory doing the analyses.
4. The PE samples do provide an opportunity to verify that the laboratories can do the analyses. In fact,
laboratories which had false positives along with low recoveries in the last round were requested to
withdraw from the program, practice their techniques and rerun their IPRs before reapplying for the
program.
5. I agree that there is a learning curve with all new methods and stated in the last update that it is too
early to make any judgements based on the PE data we have so far.
I agree with you that a formal method comparison study should be carefully designed and operated to allow
rigorous comparison. The primary purpose of the protozoan PE program is not for method comparison. It was
undertaken in response to a need for a PE program expressed by a number of the former ICR laboratories and
utilities.; Until there is a regulation in place which requires laboratory evaluation and certification, I believe that
the current protozoan PE program provides utilities with an indication of what they can expect when contracting
with laboratories to do protozoan analyses.
As laboratories continue to become more experienced, I hope to see an indication of improved recovery and
precision in the PE program. We would like to continue to provide summary data referenced to the method used.
I believe this information will be helpful to utilities trying to choose a laboratory and a method. However, your
points are well taken and I will try to be more clear in the future about what these data represent.
Thank you for your comments and I appreciate your continued support of our efforts to provide this much
needed service to the laboratory and utility community.
Sincerely,
Mary Ann Feige
-------
TSC Staff Stuffing Envelopes to Keep ICR Running -
Occasionally it becomes necessary for TSC staff (some
pictured here) to do a mass mailing (usually on a short
turn-around basis) to the ICR utilities and/or labs to keep
them aware of a new requirement or request. At times like
these we like to make it somewhat FUN by setting up a
production line complete with snacks. So, if you have
ever noticed that the envelope is a bit sticky that's
probably just the glaze off of the donuts or coffee cake.
United States
Environmental Protection Agency
Office of Ground Water and Drinking Water (MS-140)
Cincinnati, OH 45268
Official Business
Penalty for Private Usa
$300
EPA815-N-99-Q01B
BULK RATE
POSTAGE & FEES PAID
EPA
PERMIT No G-35
------- |