U.S. Environmental Protection Agency Science Advisory Board Final Minutes of Public Conference Call Meeting August 5, 2004 Committee: Contaminated Sites and RCRA Multi-Year Plan Advisory Panel of the U.S. Environmental Protection Agency's Science Advisory Board (SAB). (See attached Roster) Date and Time: August 5, 2004 from 2-5 Eastern Time (See attached Federal Register Notice ) Location: By telephone only. Call was run from Science Advisory Board, Cubicle 3610E. 1025 F Street Northwest, Washington D.C. Purpose: The purpose of this call is for the Panel to consider their draft report. Materials Available: A revised draft of the Panel's report Attendees: Kathleen White of the SAB Staff Office was present as were panelists: Clark, Crittenden, Dzombak, Kim, McFarland, Powers, Smith, and Thompson. Dellinger, Eighmy, Hughes, Lifset and Shaw were unable to join the call. EPA's Trish Erickson, ORD/NRMRL, Bob Dyer ORD/NRMRL Lynn Papa ORD/NRMRL, Dermont Bouchard ORD/Athens. Stephen Langel from Inside Washington Publishers was present. Summary The meeting went largely according to the agenda (attached) with some variation in order and timing. McFarland noted that the Panel needs to decide how to move forward with the report. There is some desire for a cross-SAB format for the multi-year plans. This draft strives for a new format using the former Panel draft as the appendix. He asked the Panel first to discuss the format and delay discussion of content until a little later in the call. Dzombak likes the format and thinks it is close enough to what was done in the past; he thought the DFO did a good job on the condensation. Thanks to panelist Reid Lifset, the Appendix is also in a consistent format. Smith thinks the format is good and believes that ORD will find the Appendix helpful. Thompson agreed with Dzombak and Smith, but cautioned that it was very important that the summary reflect what is in the appendix. Kim likes the format but is concerned that all the key points be captured. There were no additional comments or concerns. Because this discussion went more quickly than anticipated, McFarland moved the Panel to a discussion of the of the content. He reminded the Panel that Dale Pahl had commented on the importance placed by organizations, such as OMB, on reports like this and that it was important that the comments be consensus and in context. In the course of summarizing the points, it is possible that some important points were lost or diluted. He asked for the panelists specific concerns about language. Dzombak ------- doesn't want to drill down too deep too fast and suggested that the Panel first look at overall content, then into wording. Powers voiced a concern at the end of the July 7-9 meeting which is not reflected in the draft. The Panel is suggesting 10 percent, yet almost all of the emerging issues the Panel talked about come from RCRA while the money is likely to come from CERCLA. This is an important fact pointed out by the Agency that is not brought out in the report. Dzombak thought the Panel could address it by amending the list of potential topics for investigation to include some contaminated sites topic. Powers thinks it is up to EPA to figure out how to do this, but the Panel needs to be cognizant of their constraints. Dzombak said the panel should also be clear that these are examples. Concerning the budget, McFarland asked whether the Panel is saying the program is ambitious and the Agency should be commended for meeting its goals with limited funds or that the Agency is failing to meet its goals because it lacks sufficient funds. The former could be said for short-term goals, but not for long-term goals. Powers agrees with Thompson that they don't really have enough information to draw a conclusion. Dzombak thinks this point comes out in the July 30 draft's summary, but the resource issue is only alluded to in the cover letter. He views funding as a policy issue and that practically every EPA program is resource constrained so recommending more resources is not likely to be productive. Thompson agreed with Dzombak but thinks the Panel's recommendations should include a recommendation that the MYP explain clearly how it will focus on the projects of greatest strategic priority. Smith may draft some language on re-allocation. Dzombak has been removing the word "reduction" and substitution re-allocation. 1A Thompson said that what was written didn't tie closely to Appendix A. He doesn't disagree with what is said - he just can't track it to what is in the Appendix response to Question 1. DFO explained that's because the material in 1A doesn't explicitly answer the charge question and she tried to take what was there and build a supported answer out of it. All agreed the response should reflect what the Panel intended. Dzombak thought there were some good points. The idea that the MYP should be tied more tightly to the strategic plan comes up here and several places. He sees nothing in the July 30 draft that disagrees with the conclusions reached at the meeting. McFarland wants to be sure what is in the body of the report is not inconsistent with the appendix. The part about the STAR grants was discussed at the meeting, although it isn't in the appendix. Kim thinks this is the only section that does not mirror what is in the appendix. Smith, one of the 1A authors, thinks the July 30 draft is more positive than the Appendix. The reason it doesn't track well is that the documents keep changing. The Panel is not critical of EPA; it understands the difficulties, but he thinks there is no clear roadmap and no logical framework - for example, the goals numbers change. It was hard to read and it took them days of meeting to understand the roadmap. Thompson agrees with Smith. It was a great effort, but not a polished document. Powers agrees and finds the July 30 draft a little more politically correct and nice. McFarland thinks there is internal logic to each of the documents, the challenge is matching them to the strategic plan (which changed). He agrees the roadmap should be clearer. He wanted to be balanced. He will take this on as an action item. Thompson thinks the consensus of the group is captured in the Appendix paragraphs. Dzombak suggests keeping the first sentence and adding a "but" clause. McFarland wants to Panel to be clear about whether they think it is logical or not. Powers thinks it still looks like a glowing endorsement. Smith said that individually the MYPs provide a ------- logical plan; it's the "together" that is worrisome - with each other and with the strategic plan. Thompson and McFarland agree with Smith. McFarland and White will revise, send to Smith, iterate with his group (Hughes and Thompson) to better reflect intent of Panel and then revised appendix as needed to match. Clark observed a lot of this applies to the responses to 3A and 3B as well. 1B Crittenden was the lead that the response he and Powers wrote got sprinkled among the other responses, particularly the distinction between outputs and outcomes. Powers will propose moving a sentence from the appendix here; she will email the sentence. He would like the comment on summaries be brought out more; now this is in the response to 3b. McFarland thinks this is in the response to a different charge question. McFarland would welcome text from Crittenden and Powers. In general she thought their major points came out clearly. 1C Dzombak suggested some language about resource shifting rather than reductions. He will send other edits. Kim says that the concept of institutional risk scoping is missing from the July 30 draft. Kim will craft a change. 1D Powers thinks the simple approach gets too much attention and the desired too little, both in the appendix and in the July 30 draft. McFarland thinks he hears the second paragraph should be strengthened and embellished. Dzombak thinks the PART review stuff is important and should be brought forward. Powers will strengthen this section of the response. Dzombak thinks the response to 2B emphasizes the connection to PART. 1E Kim thinks this was well summarized, with one exception, "The Agency has no difficulty identifying potential emerging research areas." Dzombak has some tweaks. He noted the Appendix responses have been put in a consistent format. This has produced a lot of redundancy in 1E. Kim will look at this. Smith observed that the cover letter talks about 10% reprogramming, but the response to 1E doesn't. 2A A Panelist thought, with one caveat, it was generally a good summary. The MYPs did not make it clear what great collaborative efforts EPA has. Illustrative examples of partnering and leveraging should be included in the revised MYPs. The second bullet in the Appendix response to 2A dealt with re-scoping. Kim agrees it has to go someplace, perhaps 1C. Panelists seem happy with it being in 1C instead of 2B as long as it appears. Dzombak thinks this could go in letter to Administrator between reallocation for longer-term research, as an extra sentence or small para on the importance of re-evaluating priorities regularly. Kim will address in 1C and cover letter. 2B Thompson thought the summary was pretty good. Couldn't think of a better way to condense the material in the appendix. It seems to busy. There are advantages to specificity, but it is very busy and hard to follow. Dzombak thought it looked like the right list and Thompson confirmed it. Smith likes the specificity. All they are doing is identifying areas where they could consider re-allocation. Thompson suggested that, if it is kept, then there should be a reference to Appendix 2B Thompson had to leave. His final comments were that he was very impressed with the Agency for their thought processes and would like to see them commended for them. He made some positive comments about the July 30 draft. ------- 2C Dzombak brought up the resource shifting issue again. Smith liked the relationship and link back to 2C. Between 2B and 2C, all the issues are covered as the Panel discussed in July. 3A Clark thought it was a very good summary and had no changes to suggest. Other than Dzombak, neither did anyone else on the Panel. He had trouble with one sentence. 3B Dzombak had only minor editorial comments. No one else had any. Appendix Kim had a comment on Response to 2B Table 1, far right column. McFarland suggested it be deleted and the Panel agreed. At 3:50 McFarland asked that the panelists have their changes to the DFO by Wednesday August 18. She will revise, send to the Panel requesting any final changes and giving their approval (or disapproval) within a week. McFarland will review to see if any proposed changed require consideration by the full Panel. The cover letter and the response to charge question 1A are the only areas where extra care may be needed. Finalization can be done by email. At 4:00 the Panel discussed metrics. The DFO thinks everyone is struggling with this issue and it is tough. Additional wisdom would be welcome. Crittenden says that the main thing they are trying to measure with outcome metrics is what is the implementation of the ideas or technologies and what is the implementation. In the academic arena, they have the impact factor which relates to the number of citations a paper receives in the last two years. This is a good metric because it does tell people that they paper had an impact and is causing more work to occur or refining it. It is a way of showing the value of the work is recognized. The downside is that, in generally, engineers tend to cite engineers less often than scientists do and engineering papers are different than some science papers so that we find engineering papers are cited less than scientists papers. Nevertheless citations is a metric that can be considered and is fairly easy to come by. There are also numerous awards for people as well as projects. These are a measure of an outcome of the contributions of individuals. They aren't just for research, they are also for other kinds of achievements. (AAEE for example has a technology awards as does ASCE). Powers isn't sure these awards apply as well to the applied work at EPA. They are important, but not sufficient for a client-driven program. If the client is using the results of the research, that's good. She thinks about the technologies that are developed at EPA and whether or not they are being implemented in the wor.d Dzombak agrees the two measures Crittenden described are useful and that Powers hit on an important point as well - how to measure the use of technologies developed by EPA. In the summer of 2003 he brought people in to look at where we've been and where we are going in remediation. It is clear federal research has had a big impact. Not all of it is EPA, but EPA - especially the SITE program - has had an impact on technology development. The problem is the long lead time. One could look at records of decision (RODs) to see where research conducted by EPA has changed the frequency of use of different technologies, (bioremediation soil venting is one area he can think of) ------- McFarland invited the Agency to speak. The Panel would like to understand how EPA would use the metrics. Erickson spoke to EPA's interest in metrics. Increasingly OMB PART review, internal reviews, and others are asking federal people to demonstrate what they have achieved. On July 7 they showed retrospective counts they had done on technologies. Some of the trends become obvious 5-10 years down the road, which is not the time range that program evaluators respond to. They can show a trend of program deployment of research findings in the past, but it is hard to show this in two years. Their other interest has to do with language in the Panel's draft on outcomes and outputs. EPA understands that simply counting reports is not what they would like to see. The strategic targets in the strategic plan are more program oriented. She'd like to pick the Panel's brains. Powers thinks that the outcome/output issue was a long-term goal issue, rather than an APG/APM. At the APM, a report which can be contributed to state agencies is a reasonable measure for that APG, but not for an LTG. Dzombak thinks that, in the summary version of the report (July 30), it comes out pretty clearly that metrics for the 1-2 year timeframe, it will be outputs. Academic scientists and engineers do this, too. Sometimes they also have anecdotes about impacts, but the measures are of productivity or output oriented. On a one or two year timeframe, that's what people are stuck with. Perhaps the MYPs can acknowledge that tracking measurements will, of necessity, be output based, but there is long-term value to the work which can be demonstrated with information such as what she presented July 7. At the July meeting Smith was struck by the shorter-term issues being based on pull from the regions for assistance. In that case, a measure can be the number of problems you assisted the regions in solving. Industry, too, has to show what value they provide -typically by showing what problems they solved or that they solved them cheaper. If you have long-term goals with outcome-based metrics, the steps that lead there can hang off of the outcome. Dzombak said there was real merit in a measure of the assistance provided to the field - how interested are the customers in what you are selling. McFarland found the discussion valuable, especially the stakeholder satisfaction ones. Are there other ways to quantify this? Kim thinks EPA will have to use more than one metric because no single metric will satisfy all the needs. A possible metric would connect the research result to reaching the particular strategic goal (target?). To justify their research budget, many industries pick their top 5-10 research accomplishment and show that the savings that resulted from the research saved (or made) a lot of money. You have to be creative to come up with those numbers. McFarland asked for additional comments from Panel and got none. He welcomed additional comments to be emailed to the DFO for ORD. This issue will recur. Crittenden reminded people that he missed July 9 and asked if he missed any discussion of metrics that day. McFarland thought not. Erickson said the last five or so slides from the July 7 presentation on "Leveraging, Partnering" dealt with metrics. Crittenden would have heard that. Crittenden told the story of a Monsanto scientist who was working on biopolymers and, when he ate lunch, noticed that they were sweet, ------- which added to a new class of sweeteners. Sometimes in research there are greater outcomes than what you started out to work on. McFarland asked the DFO to re-send the slides for that briefing. The DFO had neglected to send the Agency's comments on the July 9 draft to the Panel, so she sent them during the call. Erickson indicated that the Agency would like some time to comment on the July 30 draft. The Panel settled on August 11 and extended deadline for Panel's comments to DFO to August 18. The DFO hopes to turn the draft around by August 25 and to have responses from the Panel by September 1. The DFO noted that, after the Panel has approved its report, it will need to be reviewed by the Quality Review Committee (QRC) and the Board. The meeting was adjourned at 4:45 Respectfully Submitted: Certified as True: Environmental Engineering Committee Attachments (paper) 1. Federal Register Notice 2. Agenda for the meeting 3. Committee roster 4. email approving minutes Is/ Is/ Ms. Kathleen White Designated Federal Official Dr. Michael J. McFarland, Chair CS & RCRA MYP Advisory Panel ------- |