UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON B.C. 20460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
May 28, 2008
EPA-SAB-08-009
The Honorable Stephen L. Johnson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460
Subject: Consultation on EPA's Office of Water Methodology entitled, "Drinking
Water Program Health Outcome Based Performance Measures for Chemical
Contaminants andMicrobial Contaminants (Measures Document)"
Dear Administrator Johnson:
The EPA's Office of Water (OW) requested that the Science Advisory Board (SAB) conduct
a consultation to provide input on their methodology entitled, "Drinking Water Program Health
Outcome Based Performance Measures for Chemical Contaminants andMicrobial Contaminants
(Measures Document)." A consultation is conducted under the normal requirements of the Federal
Advisory Committee Act (FACA), as amended (5 U.S.C., App.), which include advance notice of
the public meeting in the Federal Register. Although there will be no consensus report from the
SAB as a result of this consultation, the panel would nonetheless like to underscore several key
points that arose in the conduct of its consultation on the Measures Document.
On April 2, 2008, the panel met via telephone conference where representatives of EPA's
Office of Water (OW) offered an informative presentation to the members of the SAB Drinking
Water Committee (DWC). The focus of the presentations by EPA representatives for this
consultation was on the new performance measures developed by OW to link Drinking Water
Program actions to public health outcomes. A copy of the overview of the plan and the charge
questions to the committee are enclosed (Enclosure 1). In brief, the SAB was asked to comment on
the appropriateness and adequacy of using measures based on reduced bladder cancer and
cryptosporidiosis cases for chemical and microbial contaminants, respectively. EPA representatives
have stated that the new methodology is intended to be used as a tool for EPA to finalize new
Strategic Plan measures and as a future reference in implementing measures and in evaluating
progress toward measure targets. Written feedback on the charge questions was provided by several
committee members and a compilation of their comments and recommendations is enclosed
(Enclosure 2).
-------
The Committee would like to express their thanks to the presenters for their expertise,
perspectives and insights. Their contributions greatly increased our understanding of the Agency's
current policies, methods, practices and future directions proposed for linking DW programs to public
health outcomes. The DWC would like to offer some suggestions for improving the methodology.
Some individual comments are enclosed to this letter and highlighted below are several key messages
that emerged among the committee members as a result of the Agency presentations and discussions:
The Committee is supportive of the plans to link drinking water regulations to public health
outcomes and believes that this is a worthwhile and important endeavor. This first step in
developing these approaches addressing one chemical (bladder cancer and DBFs) and one microbe
(Cryptosporidium) is reasonable. However, it must be kept in mind that these are indirect measures
(there will be no direct measures of bladder cancer reduction or prevention of waterborne
cryptosporidiosis) and that there are a number of uncertainties that need to be articulated so that the
program is transparent. The Committee had a number of recommendations:
• The dose-response curve previously generated for bladder cancer risk vs.
concentration of disinfection by-products (DBFs) should be added to the appendix, or
even preferably to the text to make it clear where the estimates came from as this is a
key piece of data that will be used to calculate how many bladder cancers will be
avoided with the new rules. In addition, an explanation of how population increases
were handled in the estimates for the future and cases avoided should be made clear.
• The uncertainties in the estimates and ranges, the use of the 95% confidence limits
should be explained in the document.
• The current national standard for total trihalomethanes (TTHM) and their actual
national average occurrence should be included in the document with some
discussion of how these are related to each other and generated (one being a
maximum contaminant level (MCL), the other being a measure of how well the
utilities are doing in regard to treatment).
• For Cryptosporidium, the uncertainties in using the 1997-2000 baseline information
on occurrence should be described. Particularly as in the future the update of this
data set may be influenced by non-point sources (animal waste) climate, and rainfall
events, which may be key factors in the occurrence.
• Only existing treatment is used to address Cryptosporidium removal and this may
need to be updated and removal distributions need to be better addressed.
• Use of both the annual and cumulative measures are appropriate; however, the annual
measure could be highlighted in lieu of the cumulative measures because a
cumulative measure increases the uncertainties seen in an annual measure, and is
more uncertain. In addition, a cumulative measure is more difficult for the general
public to understand.
• EPA should be developing a plan to obtain compliance monitoring data by 2012 in a
centralized way. Resolution of the data and also small community data should be
captured.
• Newer and better metrics should be pursued so that public health benefits can be more
directly measured in the future. This may include research into how data from the
cancer registries or improved detection/reporting of endemic waterborne disease
could be measured.
• Performance measures for the Groundwater Rule should also be pursued in the future.
-------
Finally, the SAB commends the Agency on seeking early advice to enhance their
methodology for linking DW programs to public health outcomes. We look forward to working with
the Agency as they revise and implement this methodology.
Sincerely,
Joan Rose, Chair
SAB Drinking Water Committee
Enclosures
-------
ENCLOSURE 1 - Memo transmitting Charge Questions
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C.
Itti I 0 ffllS
°5Sr
SUBJECT: Request for Science Advisor* Board (SAB) Consultation on Measures to
Link.J^rinking Wdtcr Programs to Public Health Outcomes
'-•"'" K .^--:'"Z f-J < + 'i
.r— *— l -••••? -jv-- *
FROM: Nanci Gdt>, Deputy Dirojior
Office of Ground Water and Drinking Water
TO: Sue ShallaL Ph.D
Designated Federal Officer
BPA~$cience Advisory Board ScalYGffice (HOOF)
'I he Office of Ground Water and Drinking Water is requesting a consultation by the
Science Advisory Board on the draft document emitted "Drinking Water Program Health
Outcome Based Performance Measures for Chemical and Mierobial Contaminants." This
"Measures Document" describes the Agency's approach for linking drinking waler
programs to public health outcomes.
In order to develop the perlbrrnuiEi- meaHurra. the Office uf Ground Water and Drinking
Water considered several approaches, the availability of data, and input ii'otn the National
Drinking Waier Advisory Council (MOWAC). Tlie new are on
bladder catsccr and cryptosporidiosi* casts for chcmicnl and microbisil contaminatts,
Msptsctively. Since the measures are based on methodologies used in line Stage 2
Disinfection Bvproflucls Rule (for reduced bladder cancer cases) and Long Term 2
Fiiiiiiiiccd Surface Walcr Treatment Rule (for reduced er\ptosporidiosis cases), the Ofllcc
is seeking SAB'a consul talion an the applii'ttlkni of these niethudologics for m erasures
purposes. bPA will cstii:ia.tc oils-year tarytl Hieasuits in 2008 and compare these
ine-asures to actual esses avoided in 2014.
Attached are the cJwi;e to the Science Advisory Board, which identifies the questions
and issues we would like SAB to address, Uic Measures Docunieul, We have alio
included the NDWAC recomnicrsdaiJons to EPA on Perfomtaiico Measures to pruS'idc
additional baekcioiine material tbrthccotisultattoir
Attaclmtents
- Charge
Measures Document
- NDWAC" Recomniimdalians lt> EPA
-------
Background and Request for SAB Drinking Water Committee Consultation on
Measures to Link Drinking Water Programs to Public Health Outcomes
The Office of Water (OW) has developed new performance measures to link Drinking Water
Program actions to public health outcomes. After considering several possible approaches, data
available to support potential measures for a number of regulated contaminants, and input from the
National Drinking Water Advisory Council (NDWAC), OW has developed two performance
measures. The measures are based on reduced bladder cancer and cryptosporidiosis cases for
chemical and microbial contaminants, respectively. OW will estimate the annual cases that will be
reduced for each measure in 2014 and include these target estimates in EPA's 2008 Strategic Plan.
The chemical and microbial measures are based on methodologies used in economic analyses for the
Stage 2 Disinfection Byproduct Rule (Stage 2) and Long Term 2 Enhanced Surface Water Treatment
Rule (LT2), respectively. The Office of Water is not requesting consultation on these methodologies
because they have previously been peer reviewed. The attached draft Drinking Water Program
Health Outcome Based Performance Measures for Chemical Contaminants and Microbial
Contaminants (Measures Document) provides 1) language describing the measure metric, 2) a
summary of the measure methodology, 3) measure baseline metrics (i.e., "Pre-Rule" baselines) 4)
"Post-Rule" measure predictions or targets, 5) the methodology the Agency plans to use for
calculating "Post-Rule" measure metrics in the future when data are collected and 6) a discussion of
uncertainty (i.e., modeling and model-based measure metrics). The document was developed as a
tool for EPA to use in finalizing new Strategic Plan measures and as a future reference in
implementing measures and in evaluating progress toward measure targets.
OW has asked the SAB Drinking Water Committee to hold a consultation on this document.
Specifically, OW is interested in feedback from the SAB Drinking Water Committee on the
following general questions / topics related to the draft Measures Document:
1) Please comment on the discussion of the program measures (i.e., health outcomes and relationship
to regulatory program activities) and model / measure uncertainty. Does the discussion provide the
reader with a balanced understanding of the measure's value in assessing program results?
2) In order to use actual compliance data to calculate the cases avoided by 2014, EPA will need to
work with States to obtain drinking water data not available at the federal level. Practically, EPA
envisions using data collected through 2012 to allow time for analysis in completing a "Post-Rule"
measure by 2014. The NDWAC recommended that EPA continue to work with States to institute an
ongoing data-sharing process to support current and future drinking water rules and any related
performance measures. Please identify other data sources besides state compliance data that EPA can
draw on for the 2014 analysis.
3) OW has also included a cumulative measure as well as an annual measure. Please comment on
which of these indicators might serve as a better indicator of public health outcomes and include the
basis for the selection.
In addition to the above charge on the document, please identify issues for further research that
could support development of other measure approaches as the Stage 2 and LT2 rules are fully
implemented, as well as any future program measures that the Agency should consider developing.
-------
ENCLOSURE 2 - Individual Panel Member Comments
Dr. Mark Borchardt - email dated 4/2/08
Charge Question 4
Regarding the question on "any future program measures that the Agency should consider
developing." One of my concerns, perhaps unjustified, for the two performance measures selected is
that EPA will spend effort on improving these measures at the risk of ignoring those community
water systems (CWSs) that do not fall under the Stage 2 DBF or LT2 rules. I'm referring specifically
to the 18,000 or so CWSs that use groundwater and do not disinfect. These systems currently fall
under the Groundwater Rule and TCR. Offhand, I can't think of a performance measure that could
be derived from these rules, but I recommend EPA give this some consideration. Ideally, the suite of
performance measured selected should reflect EPA's responsibilities to all CWSs in the nation.
Dr. Joseph Landolph - e-mail dated 4/8/08
I very much enjoyed the telephone conference call with you both and all our DWC Colleagues on
the document entitled, "Drinking Water Program Health Outcome Based Performance Measures for
Chemical Contaminants andMicrobial Contaminants."
In addition to the discussion, I have a few comments to be transmitted to EPA.
1. I recommend that they make it very clear that they are using a dose-response curve previously
generated for bladder cancer risk vs. concentration of DBFs. I recommend that they add such a
curve to the appendix, or even preferably to the text.
This is a key piece of data that they will use, if I understand their document correctly, to calculate
how many bladder cancers will be avoided with the new rules.
2. I recommend that they make it very clear in the text, that they are not going to actually look at
Cancer Registry Data, and measure how many bladder cancers are actually avoided, but that they are
going to use the dose-response curve for bladder cancer risk vs. concentration of DBFs, and the new
concentration of DBFs, to calculate (or take from the dose-response curve) what the new bladder
cancer risk is, and calculate the new number of bladder cancer cases, and compare it to the old
number, before the DBF levels are adjusted. This issue was not clear in the text at all to me.
3. It would also be helpful if they could show on the curve, what the present DBF levels are, and
where they will be with the new rules, to make it crystal clear for the reader how much DBF and
TTHM levels would decrease. Also, if they could add a small amount of text to indicate what will
change to make this happen, this would make the document much more transparent. I would like to
see this document be able to "stand alone" as much as possible, and to be as clear as possible.
-------
Gina M. Solomon, M.D., M.P.H.
April 4, 2008
The concept of linking drinking water quality regulations to public health outcomes is a worthwhile
and important endeavor that could help to track the success of regulatory activities. It is reasonable
for EPA to start this activity with two measures, one chemical and one microbial. It is also clear that
currently there are no direct ways to measure the public health benefits from these rules, so indirect
measures need to be devised for the time being. The models that EPA sets forth are a reasonable first
step, although they are far from ideal. It is reasonable to pursue this approach, although it will be
important to incorporate newer and better methods as they become available.
There are some problems with the proposed approach that EPA lays out in the documents for review.
The two major problems are that: (1) the proposed approach is fairly circular, in that EPA is
planning to largely use the benefit analysis used in the rulemaking to track the success of the rule.
This methodology only has a minimal role for actual measurements of success (rather than
calculations), and even that is indirect and not transparent; and (2) there are many complexities that
could seriously influence the actual benefits obtained from these rules that are not considered in the
EPA analysis.
1) Please comment on the discussion of the program measures and model/measure uncertainty.
Does the discussion provide the reader with a balanced understanding of the measure's
value in assessing program results?
The discussion of the measure and uncertainty is inadequate to convey the uncertainty and the
somewhat circular and indirect nature of the metrics chosen. EPA should add additional explanatory
sections to the chemical contaminants section to more clearly explain the following issues: (1) EPA
is not planning any direct measures of bladder cancer reduction in 2014, and instead is planning to
calculate the reduction based on compliance monitoring; (2) compliance monitoring is currently not
available to EPA or to the public in a centralized way, and EPA must explain how it intends to
obtain the data by 2012 and what it will do if it does not obtain the data from some regions; (3) a
discussion of why EPA did not assume any change in bladder cancer cases between now and 2014
despite population increases; (4) further explanation of the uncertainty inherent in the range of
estimates for bladder cancer attributed to DBFs.
In the section on Cryptosporidium, EPA should explain (1) the uncertainties inherent in the use of
old data (from 1997-2000) as the baseline; (2) the data on links to weather patterns and crypto
occurrence in source water (ie. following extreme rainfall events), and the potential that climate
change will result in higher rates of crypto occurrence in source water, including how EPA intends
to incorporate this issue into the projections; (3) the fact that crypto can occur in epidemics, in
addition to endemic cases, and that these are not addressed in this analysis.
2) Please identify other data sources besides state compliance data that EPA can draw on for
the 2014 analysis.
This exercise illustrates the overriding importance of obtaining better drinking water data on the
federal level. EPA must have access to high-quality national data on drinking water quality,
including compliance monitoring data. Furthermore, in order for the benefits assessment to be
-------
transparent, these data must also be made publicly available. Only with national tracking of
compliance data will a full benefits assessment be possible.
The other data source that EPA mentions as a future possibility is the CDC NHANES exposure
study. These data will be extremely useful for assessment of EPA drinking water regulations and
other regulations. For example, arsenic has been biomonitored in the past few NHANES surveys,
and it would be important to see if there is a decline with the new arsenic MCL. Biomonitoring of
DBFs and other chemical contaminants can also provide important information about regulatory
effectiveness. It should be a high priority for EPA to work with CDC on analyses of the
biomonitoring data to assess the benefits of regulations. Finally, tracking of water borne disease (ie.
WBDSS) needs to be improved to allow more direct measures of benefits related to pathogen
reduction in drinking water.
3) OWhas also included a cumulative measure as well as an annual measure. Please comment
on which of these indicators might serve as a better indicator of public health outcomes and
include the basis for this selection.
An annual measure is a somewhat better indicator than a cumulative measure for at least two
reasons: First, a cumulative measure essentially multiplies out the uncertainties seen in an annual
measure, and so is more uncertain without any clear benefit in exchange. Second, a cumulative
measure is more difficult for the general public to understand, since people tend to more intuitively
understand annualized numbers. For these reasons, I would favor the annual measure for purposes of
describing public health benefits.
4) Please identify issues for further research that could support development of other measure
approaches as the Stage 2 andLT2 rules are fully implemented, as well as any future
program measures that the Agency should consider developing.
Some important areas for further research include:
• Assessing bladder cancer risk from chloramination DBFs and developing a delta between the
risk from chlorine disinfection and chloramine disinfection byproducts. This delta will be the
most appropriate metric for assessing the benefits of the Stage 2 rule.
• Developing an informatics system to obtain and manage compliance monitoring data from
the states, so that EPA and the public have access to the data that allows assessment of
effectiveness of existing rules.
• Working with CDC to incorporate biomonitoring measures for relevant chemicals into
NHANES and to analyze the results over time as a metric of efficacy of exposure reduction
efforts. Such analyses can already be done for THMs and arsenic, at a minimum.
• Working with CDC to improve the Water Borne Disease Surveillance System so that
improved direct measures of microbial illness will be available in the future for tracking
disease in the population.
-------
Dr. Christine A. Owen
April 9, 2008
1. Does the discussion provide the reader with a balanced understanding of the measure's
value in assessing program results?
The discussion is lacking balance and clarity in a number of aspects. The most significant
issue for EPA OGWDW staff is that the measure they have identified is a "metric" which can only
be estimated indirectly. To fully appreciate and understand this point it is important to refer back to
the EA's developed for the rule suite; this concern is particular relevant to the LT2 discussion. The
object intent of LT2 is to reduce illness (and morbidity) due to exposure to Cryptosporidium. We
currently do not have a robust method of tracking this illness in the US and so have to estimate the
baseline number of cases. To do this, EPA has created several methods to estimate the occurrence of
protozoa in source waters (based on the results of various source water monitoring efforts and
microbiological methods); once this occurrence estimate is established, then a treatment efficacy
factor is applied. The treatment efficacy factor is derived from various studies; it is an estimate of
how well protozoans are removed or inactivated by different treatment processes. At this point,
EPA then has an estimate of exposure. Exposure is then translated into illness by assuming that the
numbers of oocysts present in finished water are evenly distributed between the five strains of
Cryptosporidium about which we have infectivity information. From there, GI illness is projected.
Essentially we have created a baseline measure of disease occurrence that is predicated on a series of
estimated factors. Understandably this may well be the best estimate that we can develop at this
time. However, the issue becomes more problematic when it becomes clear that the future case
(which is the measure of the public health improvement we expect from the rule implementation) is
again the product of all the same estimated factors identified above. This ability of this metric to
measure changes is hampered by the inherent uncertainty in the estimates. To that I would add the
additional concern that it is unclear to me how this proposed metric will be able to identify
statistically significant differences between the estimated current baseline condition and the
estimated future condition.
I understand that there are limitations to what we know now; I would propose that EPA OGWDW
should invest in developing metrics which come closer to measuring the health outcome we are
trying to protect. If we start now to develop the surveillance tools to measure illness or exposure (or
some related health point) then in ten years we have data that will help assess the success of our
regulatory framework.
I recognize that assumptions are necessary in developing these metrics; however, we have to
acknowledge their shortcomings and we should focus on finding metrics that are more directly
related to the endpoints we are trying to avoid (illness, death, cancer, etc.) I have the same general
concerns for the DBF exposure scenarios. The measures proposed for both rules will only be able
(at best) to provide a sense of what can be accomplished by 2014 and only if one accepts the
estimated baselines and outcomes.
2. Please identify other data sources besides state compliance data that EPA can
draw on for the 2014 analysis.
-------
Based on my experience with the federal and state data collection and storage
systems, I am not sure that there are many other opportunities currently available for
EPA OGWDW to access. This is a most opportune time to strongly urge the Agency
to initiate activities to better coordinate with the states and other agencies such as the
CDC to develop the survey/surveillance/monitoring/reporting tools to gather the data
and information needed to identify and utilize good metrics. One suggestion to the
Agency could be that they propose measures that could indicate
progress/improvement on a wider program basis. Although our regulatory
framework has taken a contaminant by contaminant like approach, health endpoints
appear to overlap.
3. Cumulative vs. annual measure: Please comment on which of these metrics
might serve as a better indicator of public health outcomes and include the basis for
the section.
The measures proposed should be able to span both annual and cumulative impacts. Relevant health
based endpoints for each rule may be interpreted and relevant as things such as annual GI illnesses
or lifetime cancer avoided as well as annual cases of bladder cancer avoided. Both timeframes are
needed.
4. Please identify issues for further research that could support development of other measure
approaches, as well as any future program measures that the Agency should consider developing.
At this point it is critical that EPA focus on developing means of gathering the most appropriate data
available to measure health outcomes that are the predicted result of rule implementation. They
should establish data collection and data storage criteria (a needs assessment) and provide a uniform
means of measuring compliance with existing and future rules. One possible means of doing this
from here forward would be to include in each rule, recommendations (or preferably requirements)
for the data elements that would be needed for compliance reporting. Standardizing reporting
formats and the relevant data elements would facilitate future analyses. An excellent case study for
establishing data consistency is the current state of compliance reporting for the Total Coliform
Rule. Ongoing data analyses have been challenging for all parties because of how data is managed
by each state.
-------
Dr. Ric Sakaji
April 6, 2008
1. In the overview there are places where the 2014 target estimates of annual avoided cases are
presented as ranges. The statement also contains a reference to a 95% confidence interval which
seems to be associated with the target estimate range. Is it correct to have a 95% confidence interval
associated with a range? Is the 95% CI associated with the range or with a single point estimate?
Also note that CI assumes a Gaussian distribution for the population being sampled.
2. The population weighted TTHM pre-LTl & LT2 is 38 ug/L, which is being used as the baseline
for the benefits measure. The discussion should include a discussion of the primary MCL (80 ug/L).
The discussion should also include a discussion indicating utilities operate to maintain DBFs well
below the primary standard. How will the public view the role of USEPA in protecting public health
if the primary standard is 80 ug/L but the population weighted national average is 38 ug/L? If the
consumer's CCR show their TTHM exposure to be 60 ug/L, they will want to know if they are at
higher risk.
3. The measures document should also include a discussion on how the model will be recalibrated
as data becomes available. The document should be acknowledged as being a "living document.
4. Exhibit 7, the probability distribution for the treatment assumption, i.e., the log removal
distribution is characterized as being triangular. The y-axis is a relative probability without a "0"
point. I assume this was done on purpose, otherwise the probability distribution should look like a
Washington Monument and the area under the distribution should add up to 1. Do these figures
provide a balanced presentation to the reader if the probabilities are not enumerated?
------- |