KK? Why 2K? Because sometimes one K is just not enough. - Kris Kringle ICR Update Jim Walasek, Editor Technical Support Center December 31, 1999 Special Millennium Issue ICR Update ISSUe Number 20 -This information sheet, the ICR Update, is the twentieth one to be issued by the Technical Support Center (TSC) of the Office of Ground Water and Drinking Water (OGWDW). Future issues (if any) will be distributed as needed to maintain information flow related to the ICR. Editor's Note: Recently, when I've spoken to ICR Technical Contacts at water utilities affected by the ICR they almost always express relief that the ICR is finally "wrapping up" after all these years. Remember the rule was proposed in February of 1994 and then the final ICR rule was promulgated on May 14,1996. Most of the utilities received their "applicability" letters back in June of 1996. Sampling began in July of 1997 and here we are about to enter the new millennium (I know, it's not really the new millennium until 2001, but the big parties will happen as we enter 2000) with 18 months of ICR sampling completed and the final reports on the way. In fact, eighteen months of ICR data have now been validated and verified and final reports have been sent to ICR utilities and labs. The ICR data is being used to support regulatory impact analysis and to validate the water treatment plant model. A large percentage of the TSC staff has worked on this project full time since the git-go. My involvement with the ICR actually goes back to 1994 when I worked on the development of the sampling video for protozoa and viruses plus the companion guide. In 1995 TSC developed the virus monitoring protocol video as well as the ICR Protozoan Method video (both of these also had companion guides). About this time we also began work on the videos that introduced the ICR Water Utility Database System and the ICR Laboratory QC Database System which had been under development since 1994. TSC also developed several manuals to assist the ICR community in the implementation of the ICR (ICR Sampling Manual^ ICR Microbial Laboratory Manual, DBP/ICR Analytical Methods Manual, ICR Manual for Bench-and Pilot-Scale Treatment Studies, and the ICR Reference Manual - Understanding the ICR). The TSC ------- chemistry team developed and implemented the ICR laboratory "approval" procedure (including the PE program) and provided technical assistance throughout the ICR. The same is true for the microbial coordinator. The treatment studies coordinator developed the manual, reviewed treatment study plans, provided technical assistance, developed the reporting software, reviewed final reports and even designed the database to analyze the data. Finally, our computer guru played a big role in the development of the ICR Federal database (ICRFED), developed the criteria and applicability determinations for the ICR, and ensured that the ICRFED performed as originally envisioned. So, as you can see, our involvement in the ICR here at TSC goes back many years, and, YES, we too are VERY happy to be wrapping things up. The last ICR Utility Data Validation Packages for Sampling Periods 16,17, and 18 (October, November, and December 1998) were sent out for review in November with the last set of resubmission diskettes due back just before Pearl Harbor Day. Final Reports for the last quarter of ICR data were scheduled to be sent to utilities and labs between Christmas and New Year's Day. So even with the slow down in mail delivery over the holidays they should arrive early in the new millennium assuming, of course, that there is nothing to this Y2K scare. Letters were sent out in early December to approximately 100 utilities that had not yet sent in their Final Design (SP19) diskettes. Remember, these diskettes should have been sent in with the last of the ICR monthly data transfer diskettes (end of April '99). We had requested that they submit the Final Design diskettes by the middle of December so that data could be included in the 18 month version of the AUX1 database. If you haven't sent in your Final Design diskette yet, contact me ASAP at 513-569-7919. Thanks. Well, we have finally reached the end of the ICR data gathering effort. Over the next few months we will be continuing our data correction effort by revising and testing the validation, posting, and reporting software. We will also begin to take an in-depth look at the QC data, paying particular attention to precision and accuracy. Finally, in early April, any straggler diskettes remaining will be uploaded and validation will be run for the final time. I want to personally thank all of you who worked so hard to make the ICR a success. The ICR data probably represents the highest quality data ever collected for an information gathering effort such as this. Over the 18 months of sampling, some 1.5 million analytical results were produced and submitted on over 15,000 diskettes (and those are just the ones that didn't get lost in the mail). My guess is that the hard copy reports sent out to the utilities and labs, if stacked vertically would exceed 700 feet. (I don't even want to think of the number of trees sacrificed.) Thanks again, we couldn't have done it without you! So ... What Happened to My Treatment Study Report Anyway? - Good question — I'm glad you asked! After we received the Final Treatment Study Reports in July 1999, or shortly thereafter in most cases, we conducted a thorough review of the data. Anyone that conducted a treatment study has heard from us by now and responded to our request to supply missing data or to verify questionable results. The review process has taken longer than anticipated but has resulted in the correction of much erroneous data. At this ------- point we are wrapping up the final reviews and hope to have them complete by the end of 1999. While completing the review phase of this project, we have transitioned into the analysis phase of this project. During analysis of GAC study results, we are using a logistic curve form to describe the "S" shaped breakthrough pattern observed for most DBF precursors. During analysis of membrane study results, we are evaluating both the permeate water quality as well as the productivity of the membrane process. The results of these analyses will be managed in a relational database that will be used to support the Stage 2 D/DBP rule development process. Data analysis is proceeding smoothly, and we hope to complete this phase of the project in early 2000. When we have completed our analysis, we will generate a distribution version of the ICR Treatment Study Database which will be made available to the drinking water industry. The information in this database on GAC and membrane performance will have tremendous utility far outside the context of the regulatory development, and it is your hard work that made it possible. As we come to the close of this project, I want to extend my thanks and appreciation to everyone involved in this effort. It has been a loooooong four years, but in that four years I have enjoyed interacting with you as we worked through these studies. If you have any questions regarding the treatment studies, GAC, membranes, disc golf or cycling (both road and mountain) give me a call (513-569-7131). Ask for Steve. We get letters ! - A couple of days ago Mary Ann Feige and I (the ICR Update editor) received an e-mail letter from Jennifer Clancy of Clancy Environmental Consultants regarding the protozoan PE results. I have printed Jennifer's letter and Mary Ann Feige's response in their entirety. Dear Jim and Mary Ann: I have been meaning to write to you to express my concerns about the manner in which the EPA protozoan performance evaluation (PE) results are being analyzed and communicated. I was concerned about the use of the PE data when we received the September 1999 report from Mary Ann and am even more concerned to see how the data were portrayed in the ICR Update from TSC, which treats the PE program as a de facto method comparison study. The coversheet sent by Mary Ann with each round of results states that "It is not a laboratory evaluation program" and "that there are no pass/fail criteria." This is true. As I recall, the purpose of the EPA PE program was to provide independent PE samples for laboratories to use to develop and maintain their own QC database. My understanding was that laboratory-specific results without the data summaries and method comparisons were to be sent to individual laboratories to enable them to assess their performance and identify potential performance issues. Instead, the PE results from a wide range of laboratories, many of whom began analyzing protozoa samples only in the last several months, are being summarized and analyzed to draw conclusions regarding the relative performance of Method 1623 and the ICR Method. The data collected through the PE program are unsuitable for use in a valid method comparison study for the following reasons: ------- 1. There is no control over how the PE samples are used. We have found that it is critical to analyze these samples upon receipt as storage affects the quality of the (oo)cysts. 2. Laboratories may or may not be following the method exactly as described. There is a great tendency in protozoan analyses to improvise and tweak the method. Since these data are not used for compliance and hence do not require strict adherence to the method, it is likely that laboratories do take liberties with the method. 3. Any laboratory, regardless of skill level, can participate. Many of the laboratories have never been part of a protozoan approval process including on-site visits so the quality of their operation is in question. Remember that several of the laboratories with the longest experience in protozoan testing failed the ICR approval process TWICE. Even if laboratories submit IPR results to confirm that they can perform the method acceptably to enter the program, there is no way to confirm that the IPR data are legitimate. 4. The ability to properly identify (oo)cysts is critical, yet it is apparent from the false positive results that those laboratories either have contaminated their samples or cannot properly identify the protozoa from the other confounding objects which occur commonly. 5. Laboratories analyzing samples using the ICR method tend to be those with extensive experience in protozoa analyses before and during the ICR, while most of the laboratories who are just now beginning to perform protozoa analyses are using Method 1623. There is a learning curve with Method 1623, even if one has previous experience with the ICR Method. To do a proper method comparison study, laboratories would be pre-qualified as they were to participate in the 1622/1623 validation studies by prior on-site evaluations and a qualifying PE data round. This would include initial and ongoing on-site audits to confirm laboratory performance and expertise, with particular attention to microscopy skills. It is understandable why EPA is not undertaking this exercise. However, a valid method comparison study would isolate the variable of interest the relative performance of each method from other confounding variables, such as vastly different laboratory experience, laboratory-specific modifications and tweaks to the protocol, different spike doses, etc. By providing the summary data and comparing the two methods, EPA has made this into a methods comparison study. The title of Jim's piece in the October 1999 ICR Update is "The Jury's Still Out" and goes on to describe that the two methods show no significant difference. If this were truly the case, it makes me wonder just why we needed Methods 1622/1623. These reported ICR Method recoveries are quite different than those seen in the ICR under the EPA's own approval program. Although the use of the PE program data for method comparisons may be an effort to provide all the data possible to laboratories to review, this is an improper use of the data generated from these PE samples. I would ask that EPA seriously reconsider the current approach of summarizing individual and largely incomparable laboratory PE sample results and packaging them as method comparison data. The data generated through the PE program do not constitute a method comparison study, and should not, under any circumstances, be construed this way. Despite these serious concerns, I laud EPA's efforts and initiative in implementing the protozoa PE program, which fills a very real void for laboratory performance verification, now that the ICR has ended, and laboratory certification is not available. Sincerely, Jennifer L. Clancy, Ph.D. President CLANCY ENVIRONMENTAL CONSULTANTS, INC. ------- Mary Ann Feige's response: Dear Jennifer, Thank you for your letter. I understand your concerns and would like to address each one. 1. In the PE study, laboratories are required to provide bench sheets along with their results. All holding times in the methods must be met for the analysis of PE samples; If holding times are not met, the data are not used in the overall summaries. 2. If laboratories are not following the method exactly, we have asked them to send us their deviations or if they are using a method other than 1622/1623 or the ICR method, we have asked them to send us a copy of their method. It is true that laboratories may not be following the methods exactly, but there is no guarantee that this would be the case under any program. Even in the ICR, we identified some deviations which were discovered during the follow-up audits. 3. Although we do require laboratories to provide us their initial precision and recovery data (IPR), it is true that many of the laboratories are relatively inexperienced, and this was noted in our summary and our article in the last update. The only way to confirm that the IPR are legitimate is to observe the laboratory doing the analyses. 4. The PE samples do provide an opportunity to verify that the laboratories can do the analyses. In fact, laboratories which had false positives along with low recoveries in the last round were requested to withdraw from the program, practice their techniques and rerun their IPRs before reapplying for the program. 5. I agree that there is a learning curve with all new methods and stated in the last update that it is too early to make any judgements based on the PE data we have so far. I agree with you that a formal method comparison study should be carefully designed and operated to allow rigorous comparison. The primary purpose of the protozoan PE program is not for method comparison. It was undertaken in response to a need for a PE program expressed by a number of the former ICR laboratories and utilities.; Until there is a regulation in place which requires laboratory evaluation and certification, I believe that the current protozoan PE program provides utilities with an indication of what they can expect when contracting with laboratories to do protozoan analyses. As laboratories continue to become more experienced, I hope to see an indication of improved recovery and precision in the PE program. We would like to continue to provide summary data referenced to the method used. I believe this information will be helpful to utilities trying to choose a laboratory and a method. However, your points are well taken and I will try to be more clear in the future about what these data represent. Thank you for your comments and I appreciate your continued support of our efforts to provide this much needed service to the laboratory and utility community. Sincerely, Mary Ann Feige ------- TSC Staff Stuffing Envelopes to Keep ICR Running - Occasionally it becomes necessary for TSC staff (some pictured here) to do a mass mailing (usually on a short turn-around basis) to the ICR utilities and/or labs to keep them aware of a new requirement or request. At times like these we like to make it somewhat FUN by setting up a production line complete with snacks. So, if you have ever noticed that the envelope is a bit sticky that's probably just the glaze off of the donuts or coffee cake. United States Environmental Protection Agency Office of Ground Water and Drinking Water (MS-140) Cincinnati, OH 45268 Official Business Penalty for Private Usa $300 EPA815-N-99-Q01B BULK RATE POSTAGE & FEES PAID EPA PERMIT No G-35 ------- |