\ Atmospheric Sciences Research Laboratory rj? CHEMISTRY, MATHEMATICS, METEOROLOGY, MODELING, PHYSICS PEER REVIEW REPORT ON THE IN-HOUSE RESEARCH PROGRAM BY THE METEOROLOGY DIVISION September 15-18, 1987 Panel Ernest M. Agee, Chair Raymond J. Deland David EmmJtt P.K. Misra Anthony R. Olsen Anthony D. Thrall Office of Acid Deposition, Environmental Monitoring and Quality Assurance Office of Research and Development U.S. Environmental Protection Agency Research Triangle Park, North Carolina 27711 ------- U.S. ENVIRONMENTAL PROTECTION AGENCY ATMOSPHERIC SCIENCES RESEARCH LABORATORY PEER REVIEW AND WORKSHOP MANAGEMENT SERVICES Contract Number 68-02-4129 Project Officer Ronald K. Patterson Prepared by Research and Evaluation Associates, Inc. 1030 15th Street, N.W., Suite 750 Washington, D.C. 20005 (202) 842-2200 727 Eastowne Drive, Suite 200A Chapel Hill, N.C. 27514 (919) 493-1661 ------- TABLE OF CONTENTS CHARGE TO REVIEW PANEL 2 GENERAL OVERVIEW 3 PROGRAM ASSESSMENT 6 Boundary Layer/Turbulence Research 7 Regional Oxidant Modeling 8 Regional Particulate Modeling 11 Regional Acid Deposition Modeling Area 12 Complex Terrain Dispersion Modeling 14 Wake Effects Research 15 Technology Transfer/Application 16 APPENDIX A: Peer Review Panel 18 APPENDIX B: Agenda for the Peer Review of the In-House Research by the Meteorology Division 20 APPENDIX C: Comments on Model Evaluation 26 APPENDIX D: ASRL Process Evaluation for Meteorology Division In-House Research Peer Review 29 APPENDIX E: Results of the EPA Participant Survey Meteorology Division 33 APPENDIX F: ASRL Staff Response to Reviewers' Comments 37 APPENDIX G: The Laboratory Director's Review Comments on the Panel Report and the ASRL Staff Response 49 ------- EXECUTIVE SUMMARY The Review Panel has reached a consensus on the following findings and recommendations: • Leadership of the Division is highly effective t Scientific staff is competent • Scientific morale is good f Experimental fluid dynamics is excellent • Computational quality control of model development is exemplary • Computer resources are inadequate • External pressures are forcing premature release of preliminary scientific findings • In-house statistical expertise should be strengthened to support model evaluation t A program for advanced research model implementation is strongly endorsed ------- SECTION 1 INTRODUCTION This is a report from the Peer Review Panel (see Appendix A for list of panel members) organized to review the in-house research activities of the Meteorology Division (MD) of the US Environmental Protection Agency (EPA), Atmospheric Sciences Research Laboratory (ASRL). Prior to meeting with the MD staff, the panel received three volumes of selected research papers by the MD staff for preliminary review. On September 15-18, 1987, the panel met at the Research Triangle Park, NC, to hear 45 presentations by the MD staff on in-house activities. Presentations focused on individual scientific contributions to EPA programs rather than a review of activity as project officers or monitors of extramural contracts. The panel appreciates the efforts of the MD Director and individual staff members in preparing well-organized preview material and informative presentations. In general, the quality and organization were equivalent to that of a scientific professional meeting. In organizing this report, the panel chose to follow the general scientific areas that were used to organize the presentations (see meeting agenda in Appendix B). Finally, the panel is grateful to Charlotte Coley (Research and Evaluation Associates) and Ron Patterson (ASRL Peer Review Coordinator) for assistance with the logistics in conducting the review and preparing this report. ------- SECTION 2 CHARGE TO REVIEW PANEL The panel was provided with the Division Director's preview materials outlining the mission of the Division's program. Principles to guide the peer review process were stated in the Research and Evaluation Associates, Inc. document Guide for Participants in the EPA- ASRL Task/Project Peer Review Process provided by Charlotte Coley. Further discussions with the MD Director at the beginning of the site visit helped to clarify the charge to the review panel. Evaluation of in-house research and support activities given in this report are based on the following program objectives: t Basic Research • Model Development • Applications and Evaluations The panel has strived to evaluate the research presented without the benefit of some important information such as CVs that describe and quantify total up-to-date professional activities. Although the panel was provided with detailed individual workplans, insufficient time was available to ingest all this information. The panel attempted to evaluate the in-house activity, realizing that a sizeable portion of the comprehensive program effort resides within the extramural community. Further, the panel has taken into consideration that MD program activities (although residing within the ORD) must respond to regulatory needs. ------- SECTION 3 GENERAL OVERVIEW The MD programs are conducted in a favorable in-house research support environment. Generally, the scientific staff seems to be enthusiastically engaged in their efforts, and morale within the Division seems good. Administrative leadership is effective in promoting this favorable professional environment. MD programs are an important part of both national and international research efforts, and the reputation of the staff brings respect and requests for applying in-house scientific expertise to help address the needs of EPA's policy analysis. The experimental fluid dynamics is excellent and goes far in advancing the MD program objectives. The computational quality control of model development is exemplary and in itself represents an ideal model for QC assessment that could be transferred outside the Division. However, computer resources are inadequate for model development, applications, and evaluation. Computer cycles seem to be highly variable and inadequate, and thus introduce some degree of uncertainty in carrying out the MD programs. Small computers are augmenting computing needs, but a total spectrum of computing support is required (from small work stations that stand alone or access other machines to supercomputer model simulations for numerically intensive model development and testing). In-house statistical expertise should be strengthened to support model evaluations. The activities known variously as "model evaluation" and "model validation" deserve more attention by environmental researchers, in general, and thus by the MD in its role as a leader of environmental research. There seem to be essentially two purposes of these activities: (1) to better understand atmospheric processes, and thus to improve atmospheric models, by identifying and diagnosing failures in model performance, and (2) to quantify the reliability of models that are used either to decide regulatory issues ------- or to guide regulatory policy. (See Comments on Model Evaluation given in Appendix C.) The Meteorology Division seems to be a frequent recipient of requests from other offices and laboratories within the EPA that require redirection of program FTEs. The Division appears to be responding to these requests, but at the cost of delaying longer term research needed by the Agency. In some instances (e.g. the requests to run the ROM simulations for EPA constituents), scientific expertise is steered away from model development to application runs. Some of these "quick turnaround" demands for staff time seem to be appropriate uses of expertise within the MD; however, the balance of short-term and long-term activities should be defined more explicitly. One of the major challenges to ORD Management would seem to be finding and preserving a balance of staff time allocated to quick response needs of the Agency on the one hand and longer-term needs on the other. Although a good balance seems to have been achieved by the MD, it also seems to the panel that ORD Management in general has a responsibility to both the scientific staff and to the Agency to take steps to guard against the erosion of the Agency's research capability. A recommendation of the panel with respect to this issue is to broaden the criteria used to evaluate the achievements of the scientific staff. In general, the staff members are doing an excellent job of communicating with their peers within their respective disciplines through publications and presentations. But communication across disciplines (horizontal communication) should be emphasized, for in the relatively young field of environmental research, interdisciplinary communication is less well established, although extremely important. Similarly, more of the staff should be encouraged to familiarize themselves with the Agency's use of their research to help set policy (vertical communication). This will help to ensure the relevance of individual research within the MD. More importantly, it should engender a vision and a coherent, internally developed agenda for the MD's research. Even though the MD's tasks are ultimately determined outside the Division, such an agenda would be an important reference ------- for establishing the MD's research priorities. A more active role by the MD in setting research priorities would help to guard against the erosion of the MD's research capability by an overload of quick response tasks. Finally, the emerging program concept of Advanced Research Model Implementation is strongly endorsed by the panel. Several existing modeling efforts form a nucleus for this initiative. An additional consideration should be given to nesting some of these models inside the global climate models that presently exist within the research community. ------- SECTION 4 PROGRAM ASSESSMENT To facilitate the achievement of consensus, the committee adopted an internal evaluation procedure whereby each panel member assigned the value of 5 (Excellent), 4 (Very Good), 3 (Good), 2 (Fair), or 1 (Poor) for each of the following performance criteria: • Quality of Science/Work • Relevance to MD/EPA Missions • Interaction with In-House Colleagues and Outside Community Emphasis was given to scientific accomplishment, such as reviewed journal publications and conference/workshop publications and presentations. Professional service to EPA and extramural activities were also considered. In some instances, scientific support was the focus of work activity rather than basic research or model development and testing. Recognition was also given to the importance of applied research. Although the scores are not provided in this report, the panel directed itself to write program evaluations that were consistent with composite numerical scores assigned to the performance criteria. The panel was pleased with the consistency of the individual evaluations. The reviews given below reflect more directly on some presentations than others. Some individual comments are provided, but not in detail for each individual presenter. Absence of detailed technical comments should not be construed as positive or negative. ------- Boundary Laver/Turbulence Research The ASRL is actively involved in the following areas of boundary layer and turbulence research: 1. Experimental investigation of atmospheric boundary layer turbulence, 2. Innovative dispersion model development, 3. Evaluation and assessment of dispersion models. The committee evaluated these areas of research in terms of their quality, relevance to EPA goals, and interaction of the responsible scientists with the outside scientific communities. Experimental boundary layer research includes turbulent diffusion experiments inside a convective boundary layer and urban and non-urban boundary layers. The quality of research in these areas is excellent. This research is very relevant to EPA objectives and the responsible scientists have excellent interactions with outside communities. New modeling approaches are being investigated using probability density functions of vertical velocity in convective boundary layers. For buoyant plumes, semi-empirical models have been developed. Again, the quality of research in these areas is excellent, the work is highly relevant, and there is very good interaction with outside communities. Future work in this area should emphasize better characterization of buoyant plume dispersion inside a convective boundary layer since the probability density function approach is not expected to work well. The quality of work in the application of models is very good, and the work is highly relevant and interactive with the outside community. It is good to see the efforts directed towards definition of atmospheric stability with proper boundary layer velocity and length scales. The sensitivity analyses of the models could be improved by using a joint probability density function of the variables rather than varying them independently. The rotary spectral density application in the analyses of turbulence data is not obviously relevant to EPA objectives even though ------- the quality of the work is good. Such work may not enhance our understanding of boundary layer turbulence significantly. The interaction with the outside community is not very apparent. Similarly, the analyses of turbulence data using superposition of autocorrelation functions is not very relevant. The quality of the work is fair and interaction with outside communities not very good. Eulerian autocorrelation functions do not determine diffusion in a turbulent flow. Relating Eulerian autocorrelation functions to Lagrangian autocorrelation functions is not simple and unique. Also, "true" Eulerian autocorrelation functions in the atmosphere exhibit oscillation about the zero line thus making the estimate of time scales questionable. Regional Oxidant Modeling This effort within the Division epitomizes the full spectrum of program objectives ranging from modeling development to applications and evaluation. The nucleus of staff spearheading this effort is highly competent and is attempting to entrain appropriate activities throughout the division. It seems that external pressures (outside of ASRL/MD) are responsible in part for this program initiative. These pressures also require model results that are difficult to achieve. Imposed resolutions in the Regional Oxidant Model (ROM), as required to be practical, are inconsistent with routine meteorological data sets. For example, upper air weather observations are taken every 12 hours on a mean spatial scale of 325 km. Model integrations are for one hour over an 18x18 km grid. The ROM appears to be a zero-order model in atmospheric dynamics (at best), whereas the chemistry of the 28 reactive species is treated in a more sophisticated manner. Further, the raw input data of emissions are rather discretized and irregularly available in space and time. The ROM is a pure diagnostic model, i.e. output concentrations are used (after the fact) to evaluate the effects of alternative strategies for managing air quality planning and standards emission. It is not realistic to impose point concentrations (observations and/or standards) on regional model plume dispersions. 8 ------- Skill scores and/or performance evaluation may objectively rate the ROM concentrations satisfactorily when instances of point concentrations actually fail miserably. New methods of evaluation should be considered that weigh both spatial and temporal scales or features. If the ROM is labeled as a zero-order modeling effort, then its performance will be properly evaluated. More support of this effort should be provided. Also, apparently none of the field programs to-date have provided high resolution (space and time) rawinsonde data sets. Since a diagnostic model approach is acceptable, more emphasis should be placed on collecting a more representative data set for model development and performance evaluation. ROM development apparently needs more computing cycles. Algorithms for the model are not fully understood (for example, incorporation of the effect of clouds and radiative processes). The ROM model should interact more with numerical weather prediction model efforts. Can some nested grid simulations and use of trajectory model forecasts (by NOAA-NWS) be considered? Trajectory models combined with chemical reactions code may yield some useful results. In several instances the scientific capability of the involved scientists exceeds the scientific credibility of ROM. For example, it has been hypothesized that receptor sites can be grouped into classes, such that concentrations averaged over all sites in a given receptor class at a given hour is a quantity that is approximately the same for all realizations of the concentration ensemble. If this is true, then with respect to the receptor class average (RCA), the concentration field is a quasi-deterministic variable. Therefore, model simulations of single realizations can be interpreted in the conventional deterministic manner. Although this hypothesis has not yet been tested, it is assumed for the present purposes that it is correct. It is unscientific to proceed in this way, though it may be politically and practically sound. Biogenic emissions inventory development is essential for the application of ROM. Currently, the work involves translation of known emission information to an hourly emissions ROM grid. This requires allocation of land use and biomass information to grid cells, ------- derivation of relevant biomass volumes, and identification of species emission factors. This approach appears appropriate for the intended use and necessarily involves extrapolation and disaggregation of existing information. The research has involved contact with forest and crop researchers during the development of information sources, but has not resulted in presentation of results to that community. The latter is recommended as a complement to existing reports to the air pollution modeling community. Uncertainty estimates were presented, but seemed questionable. Work has focused on ozone dry deposition over large areas as determined from aircraft measurements. Data were collected through extramural effort, but data analysis has occurred in-house. Aircraft measurements suggest that ozone concentrations may be highly correlated to organized convection. Ozone appears to be a good tracer of convection, similar to moisture. Detrended temperature should be spectrally decomposed to show the basic convective mode (BCM) or cloud street mode. An approximate 4 km length scale is evident in the raw data. The ozone spectrum presented indicates a 3.8 km spectral peak, which accords well with temperature data. What about the spectrum for the specific humidity q? Also, several papers exist in the literature that show larger length scales of mesoscale convective organization in a convective boundary layer, known as higher convective modes (HCM). The typical aspect ratio for the BCM is 3 to 1, compared to an order of magnitude increase for the larger HCMs. Inspection of the raw concentration for ozone indicates a signal on the larger length scale that is in phase with temperature trace. An effort should be made to complete the analysis of these unique data sets by interpretation of convective structures (for all fields) and treating ozone as a tracer. This has some exciting possibilities. The development and implementation of verification procedures for the ROM will significantly enhance the current and future use of ROM. Current computer science concepts have been integrated into EPA's atmospheric modeling community. This integration not only results in products of known quality, but also appears to be accomplished as a natural part of the model development. There appears to be an active 10 ------- effort to publish this approach within the air pollution modeling community. Although not directly presented during this peer review, it is known that this experience has contributed to the formulation of quality assurance procedures for the RADM model evaluation program. The presence of the Data Systems Analysis Branch staffed with knowledgeable application computer scientists is one key to the Meteorology Division's ability to fulfill their mission. Its presence allows atmospheric scientists and computer scientists to contribute their skills to an interdisciplinary effort of model development, evaluation, and application. Very systematic and methodical quality control analysis is performed with state of the art practices, from modular programming down to file naming conventions. Professional systems programming architecture could serve as a model for other similar unrelated model developments. Feedback between the physicist(s) behind the model development and programming quality control people was a little vague. Scientists that develop the model know the inherent weaknesses. Quality control should generate, independently, a similar set of weaknesses. The need for more computing cycles is not abundantly clear, but probably is the case. Funds should not be used to upgrade antiquated computing equipment. What about vectorizing the code for the ROM? Also, how much do QC checks slow down the ROM simulation? A more coordinated plan is apparently needed for providing computing facilities to ASRL/MD or even to all of EPA at Research Triangle Park. Can metacode be saved, upon which variations of model predictions can be made? Although briefly discussed, the ROM may be improved more by a parallel implementation than by a vector implementation. Also, can EPA use the supercomputer resources at the five national centers? Regional Particulate Modeling The Regional Particulate Model (RPM) program uses ROM and has some of the same problems of weak scientific support and inadequate input data, especially emissions. The work on different aspects of the model, presented by four of the scientific staff, showed evidence of 11 ------- industrious efforts to provide what is needed with inadequate resources of manpower and computing facilities, resulting in contributions that were incomplete. The cloud process module work may evolve into some good science. However, this component of both the ROM and the RPM (or any regional scale model) is not getting the attention it merits. The simplistic approach presented is a reasonable application of the state-of-the-art knowledge for simple cloud model formulation (primarily a thermodynamic emphasis). Effort needs to be extended to properly treat the total entrainment process between the cloud top and the overlying interfacial layer. Cloud ensemble scale should be considered in accordance with the ROM grid length. The TTNEPH data may be all right for radiation studies but its value in the wet deposition and cloud processing modules is very questionable. Efforts should be made to incorporate the GOES imagery, which is very useful for the ROM domain. The work on this project epitomizes the vitality offered by young new staff members. The MESOPUFFII modeling effort is probably important in principle, but (very good) performance evaluation shows poor performance. The CAPTEX data set appears adequate to test performance. Best results were obtained for cases of no directional shear. Work should be published in reviewed journals (not just an EPA report). The evaluations of the Regional Lagrangian Model of Air Pollution RELMAP has served to expose NAPAP emission inventory deficiencies. Thoroughness and comprehensive knowledge were evident in the presentation. Regional Acid Deposition Modeling Area MD in-house research activities related to regional acid deposition modeling involve the International Sulfur Deposition Model Evaluation (ISDME) project, Regional Acid Deposition Model (RADM) evaluation and assessment projects, and cumulus cloud venting based on data from VENTEX, CUVENT, NEROS, and NASA Langley field experiments. The in-house activities reviewed comprise only a fraction of the project 12 ------- activities. Hence no attempt is made to review either past or planned field experiments or the entire RADM model evaluation effort. Since the United States portion of the ISDME was completed within the MD, it will be more fully reviewed. The ISDME work has placed the MD staff at the forefront of the regional model evaluation scientific community. The ISDME Project evaluated eleven linear-chemistry atmospheric models of sulfur deposition using 1980 observational data. This effort is noteworthy as being the first organized effort at regional model evaluation. Not only has the work of MD staff contributed to an increase in knowledge on model performance, but also to new approaches to model evaluation. Although evaluation methods for regional scale models which are acceptable to all scientists have not been formally established, the ISDME scientists have employed a careful system of statistical procedures to arrive at their conclusions. The effort to rank the models' performance does not appear to aid the choice between models for policy use and requires further research to be usable in that context. In addition to the statistical evaluation, additional effort is needed to outline conditions under which the models are or are not applicable. For example, the simple Lagrangian models show the same performance repeatedly over several years when annual averages are compared. The ISDME analysis would be more useful if it had led to an understanding of why the models show this type of performance. Use of metrics based on spatial pattern is important, but requires incorporation of spatial surface interpolation. Further research into the applicabilities of kriging to surface concentration and deposition fields is recommended. Current in-house research focuses on design studies for surface network, intra-spacing network and emissions data collection. The design work relies heavily on kriging but does provide a basis for quantitative design studies that focus on minimizing the evaluation "error". The MD staff recognizes that kriging requires further evaluations as a viable design and model evaluation tool and that other evaluation methods should be explored. 13 ------- The cloud venting research focuses on the study of penetrative convection, which leads to venting of the planetary boundary layer and transport of pollutants. The results of this work have important implications for both the ROM and RADM programs and should be continued. The researcher is encouraged to establish stronger research relationships with the cloud dynamics community. The field experiment data are unique and deserve more detailed documentation of the meteorology surrounding the events. The further classification and determination of 6 factors for several types of cloud-topped (fractionally) planetary boundary layer is recommended. Overall, the MD staff involved in regional acid deposition modeling are (1) performing research that is very relevant to EPA programs, (2) actively interacting with the relevant scientific community, and (3) completing research that is worthy of publication (and is being published) in the peer-reviewed literature. Complex Terrain Dispersion Modeling Nearly 30% of the presentations were associated with in-house efforts to support the Complex Terrain Dispersion Modeling (CTDM) program. The panel was unanimous in its positive evaluation of the quality of the experimental modeling activity. The ability and willingness of the fluid modeling team to respond to the demands of a regulatory agency and still be able to carry out basic research speaks well for both the scientists and division management. Since the panel was not briefed on the complete CTDM program, it was unable to review the in-house research within the context of the whole project. However, the following specific comments can be made: 1) Good, interesting results are obtained in the fluid modeling laboratory and should continue to be published in JFM type journals. The panel, however, questions whether the laboratory experiments are sufficiently reliable simulations of atmospheric flows to be used to set regulatory guidelines. 2) Inverted tows in stratified cases are adequate, however a deep sheared flow layer is not present (as in the neutral wind tunnel case 14 ------- studies). Experimental fluid mechanics is done well, both in laboratory techniques (flow visualization and measurement) and application of theory. How these results are factored into the CTDM was not explained well. 3) Relevance of model and laboratory results (even with good agreement) do not necessarily apply to atmospheric boundary layer flows. Stably stratified flows are useful, but do not address the convective planetary boundary layers. Dynamic similarity between numerical model/laboratory flow and atmospheric analog was not established. It appears that a constant eddy viscosity value arbitrarily replaces the molecular viscosity in the model momentum equation. In reality, isotropic Fickian diffusion cannot be assumed. Further, the spatial dependencies in nonlinear systems of equations give rise to non-Gaussian dispersion. Again, the fluid mechanics work is excellent but its extension to actual atmospheric dispersion is not clearly explained. Wake Effects Research Although driven by regulatory requirements, the wake effects research can be considered quite basic. The work on the characterization of the complex turbulence in the wake of a non-aerodynamic structure is a good example. The need for an improved plume formulation is obvious and common to much of the theoretical and physical modeling efforts at MD. Efforts to obtain better flow visualization and, in particular, to quantify those observations are to be commended. Illumination techniques and the statistical evaluation of the resulting data need to be improved. New plume formulations call for a departure from the traditional Gaussian approach. Any fundamental advances in this area would have an impact not only on EPA activities but also on the boundary layer turbulence research community in general. The work on auto exhaust dispersion is clearly associated with EPA's need to model the impact of a major source of atmospheric pollution. The experimental results presented during the review were 15 ------- based upon only a part of a more comprehensive program of experiments. It is difficult therefore to judge what should or should not be done since the work described was very incomplete in itself. Technology Transfer/Application This set of presentations, along with two presented as a part of Boundary Layer/Turbulence Research, seemed to be a representative selection of work by the Environmental Applications Branch (EAB). That is, the presentations were consistent the function of the EAB. The panel regards the UNAMAP to be of very good quality and to be a most important function of the MD. Indeed, UNAMAP is one of the most important and visible aspects of the EPA's air quality operations, since regulatory applications (e.g., obtaining permits, establishing emission limits, or demonstrating attainment of air quality standards) typically require the use of a UNAMAP model. The estimation of the reduction of soybean crops resulting from various levels of ozone was presented. This analysis was requested by the Office of Air Quality Planning and Standards. Current levels of ozone at 320 farms growing soybeans were related to existing and proposed national ambient air quality standards (NAAQS) for ozone. An important finding from this research was that, contrary to greenhouse studies, field studies of the effect of ozone on soybean growth suggested that two parameters of soybean exposure to ozone (namely, the duration and the concentration of the exposure) could be summarized by a single index (effective mean ozone concentration). This work seems to the panel to be highly responsive to the needs of EPA policy analysis, and like previous work at the MD on the relationship between concentration averaging times, may be widely used and cited. Due to the importance of this work, it deserves further investigation and the collaboration of other experts in meteorology, atmospheric chemistry, plant physiology, and statistical inference. The panel rated as very important the investigation of alternative techniques for randomly sampling data to efficiently estimate long-term (seasonal or annual) concentrations using the Point, Area, and 16 ------- Line-Source (PAL) algorithm. This work addresses a practical need for computational efficiency, but, more significantly, explores probabilistic modes of air quality analysis. This work should continue and should be supported by a statistician who is an expert in the techniques and inferential issues of data sampling schemes. Finally, a preliminary diagnosis of the Turbulent Profile Sigmas (TUPOS) dispersion model based on a comparison of calculated and measured SFs concentrations obtained from the EPRI's Kincaid data base was presented. The investigation will help to determine the degree to which more detailed turbulence data improves model performance under field conditions and is therefore regarded as extremely important by the panel. As with other model diagnoses, sensitivity studies, and reliability studies conducted by the Division, the statistical issues are quite challenging (see Section 3 and Appendix C of this review) and deserve the support of an expert statistician. 17 ------- APPENDIX A PEER REVIEW PANEL ------- EPA-ASRL Peer Review Panel Meteorology Division Research Triangle Park, North Carolina Septenber 15-18, 1987 Ernest M. Agee Chairman, Dept. of Earth and Atmospheric Sciences Purdue University West Lafayette, IN 47907 (317) 494-3282 Raymond J. Deland Raymond Deland & Associates 2 Collyer Drive Ossining, N3f 10562 (914) 941-5622 David Etnmitt Simpson Weather Associates, Inc. 809 E. Jefferson Street Charlottesville, VA 22902 (804) 979-3571 P.K. Misra Ministry of the Environment 125 Resources Road East Wing Rexdale, Ontario Canada M9W 5L1 (416) 235-5771 Anthony R. Olsen Battelle Pacific Northwest Program P.O. Box 999 Richland, WA 99352 (509) 376-4265 Anthony D. Thrall Environment Risk Assesntent Program Electric Power Research Institute 3412 Eillville Avenue Palo Alto, CA 94304 (415) 855-2627 19 ------- APPENDIX B AGENDA FOR EPA-ASRL PEER REVIEW OF METEOROLOGY AND ASSESSMENT DIVISION SEPTEMBER 15-18, 1987 ------- AGENDA Peer Review of the EPA-ASKL Meteorology and Assessment Division In-Hbuse Research Program September 15-18, 1987 TIME TCPIC SPlMiK REF.f Tuesday. Sept. 15 Classroom 3, Environmental Research Center Research Triangle Park, North Carolina i 08:00 - 08:20 AM Opening Session - Panel Welcome Charlotte Coley Peer Review Program Orientation ASRL Peer Review Coordinator Ron Patterson 08:20 - 08:40 AM Executive Session in Meeting Al Ellison Room for Panel and Staff, ASRL Director 08:40 - 09:00 AM Coffee Break and Introductions of Review Panel and ASRL Staff 09:00 - 09:10 AM Introduction by MD Director Frank Schiermeier 1 Boundary Laver/Itarbulence Research 09:10 - 09:40 AM Analysis and Parameterization Gary Briggs 2 of Convective Diffusion Processes 09:40 - 10:00 AM Evaluation of Convective Scaling Tom Pierce 3 for Estimating Diffusion 10:00 - 10:30 AM Boundary Layer Turbulence Studies Jason Ching 4 in Urban and Non-Urban Areas 10:30 - 11:00 AM Rotary Spectral Analysis of Peter Finkelstein 5 Turbulence Measurements 11:00 - 11:30 AM Exponential-Sum-Fitting Tech- Steve Perry 6 niques for On-Site Turbulence Analysis 11:30 - 12:30 AM Lunch 12:30 - 01:00 PM Executive Session for Review Panel 01:00 - 01:20 PM Meteorological Scaling in Applied John Irwin 7 Dispersion Modeling 21 ------- 01:20 - 01:35 EM 01:35 - 02:20 PM 02:20 - 02:40 EM 02:40 - 03:00 EM 03:00 - 03:30 EM 03:30 - 04:00 EM 04:00 - 04:30 EM 04:30 - 05:00 EM Wednesday. Sept. 16 08:00 - 08:20 AM 08:20 - 08:40 AM 08:40 - 08:50 AM 08:50 - 09:15 AM 09:15 - 09:40 AM 09:40 - 10:00 AM Relation of Error Bounds of Maxmium Concentration Estimates to Meteorology Uncertainty Regional Oxidant Modeling Regional Oxidant Model (BCM) Biogenic Emissions Inventory Development for Regional Oxidant Modeling Break Ozone Dry Deposition Over Large Areas from Aircraft Measurements Verification Procedures Applied to the Regional Oxidant Model Evaluation of Regional Oxidant Model Executive Session for Raview Eanel and MD Division Director John Irwin Bob Lanb Jim Reagan Jim Godowitch Joan Novak Ken Schere Frank Schiermeier Particulate Modeling Regional Particulate Model (EPM) John Clarke doud Processes Module for Frank Binkowski Regional Particulate Modeling Utilization of X^NEXU. Data for Regional Particulate Modeling Russ Bullock Sensitivity Analysis of MESOPDFF Jim Godowitch n and Evaluation with CAETEX Data Regional Lagrangian Model of Air Brian Eder Polution (HELMVP) Sensitivity Analysis and Evaluation for Particulate Matter Coffee Break and Introductions of Review Panel and ASKL Presenters Deposition Modeling 10:00 - 10:30 AM International Sulfur Deposition Model Evaluation (ISDME) Terry dark 9 10 n 12 13 14 15 16 17 18 19 22 ------- 10:30 - 11:00 AM Regional Acid Deposition Model Robin Dennis 20 (RADM) Evaluation Research Program 11:00 - 11:30 AM Cumulus Venting of Pollutants Jason Ching 21 from the Mixed Layer 11:30 - 12:30 PM Lunch 12:30 - 01:00 PM Executive Session for Review Panel 01:00 - 01:30 PM Acid Deposition Assessment Robin Dennis 22 Studies Complex Terrain Dispersion Modeling 01:30 - 01:40 PM Overview of Complex Terrain Frank Schiermeier 23 Research 01:40 - 02:00 PM Complex Terrain Field Data Bases Larry Truppi 24 02:00 - 02:20 PM Fluid Modeling Facility Object- Bill Snyder 25 ivesf Facilities, Approach, and Recent Research Accomplishments 02:20 - 02:40 PM Deformation of Plumes by Hills Bill Snyder 26 02:40 - 03:00 PM Break 03:00 - 03:20 PM Measurements of Streamline Traj- Roger Thompson 27 ectories in Stratified Flow Over Isolated 03:20 - 03:35 PM Laboratory Measurements of Bill Snyder 28 Unsteadiness of Strongly Strat- ified Flow Fields CDrag) Over Two-dimensional Hills 03:35 - 03:50 PM Comparison of Numerical and Roger Thompson 29 Laboratory Experiments on Density- Stratified Flow Over a Hill 03:50 - 04:10 PM Stratified Flow Over Ridges and Bob Lawson 30 Valleys flushing of Valleys) 04:10 - 04:30 PM Laboratory Validation of Flat- Bill Snyder 31 Dividing-Streamline-Surface Approximation for Complex Terrain Dispersion Models 04:30 - 05:00 PM Executive Session for Review Frank Schierneier Panel and M5 Division Director 23 ------- Thursday, Sept. 17 08:00 - 08:20 AM Wind Direction Effects on Bill Snyder 32 Dispersion from Sources Downwind of 08:20 - 08:50 AM Complex Terrain Amplification Bob Lawson 33 Factors for Various Stack Heights/ Locations 08:50 - 09:30 AM Diffusion from Low Level Sources Bert Eskridge 34 Under Extreme Stratification 09:30 - 09:50 AM Coffee Break and Introductions of Review Panel and ASRL Presenters Wake Effects Research 09:50 - 10:20 AM Theoretical Modeling and Eval- Alan Huber 35 uation of Building Wake Effects 10:20 - 10:40 AM Observations of Surface Flow Bob Lawson 36 Patterns Near Buildings Using Flow Visualization 10:40 - 11:10 AM Video Image Techniques for Wind Alan Huber 37 Tunnel Measurements of Building Wake Dispersion 11:10 - 11:30 AM Near-Wake Dispersion of Auto Roger Thompson 38 Exhaust 11:30 - 12:30 PM Lunch 12:30 - 01:00 PM Executive Session for Review Panel Technology Transfer Applications 01:00 - 01:30 EM User's Network for Applied Bruce Turner 39 Modeling of Air Pollution CUNAMAP) 01:30 - 01:50 EM Development and Application of Ralph Larsen 40 Air Quality Effects Models 01:50 - 02:10 PM Climatological Version of the Bill Petersen 41 Point Areas, and Line Source Algorithm CPAL) 02:10 - 02:30 EM Evaluation of the TUPOS Dispersion Bruce Turner 42 Model - Preliminary Findings 02:30 - 02:50 EM Break 24 ------- 02:50 - 03:10 EM Conparing Air Quality with Ralph Larsen 43 Present or Proposed Eegulatory Standards 03:10 - 03:30 PM Rise and Dispersion of Merging Bill Snyder 44 Buoyant Plumes in a Stratified Crossflow (Ocean Outfalls) 03:30 - 03:50 PM Dispersion of Dense Gas Over a Bill Snyder 45 Ramp 03:50 - 04:15 PM Advanced Model Operation and Frank Schierneier 46 Analysis 04:15 - 04:45 PM Executive Session for Review Panel Frank Schienneier and MD Division Director Friday. Sept. 18 08:30 - 10:00 AM. Reviewer Debriefing with ASRL Al Ellison Director 10:10 AM Report Preparation 25 ------- APPENDIX C COMMENTS ON MODEL EVALUATION ------- COMMENTS ON MODEL EVALUATION It seems useful to distinguish the following four types of model evaluation: (1) model diagnosis (2) reliability estimation (3) sensitivity analysis (4) uncertainty analysis Model diagnosis is the comparison of measurements to model calculations so that inadequate assumptions or formulations embodied in the model can be identified and investigated. The investigation often leads to model improvements, so that model diagnosis is typically one phase of model development. Such diagnoses are often informal data analyses used to generate new hypotheses. Of course, if such an informal data analysis suggests a major redirection of research or model development, the diagnosis is worth checking more rigorously. Estimating the reliability of a model requires more careful and thorough statistical analyses than does model diagnosis. First, the use or uses of the model for which the reliability analysis is being conducted must be carefully defined. Then a means of estimating the model's reliability must be determined. Finally, the degree to which the results of analysis generalize to other conditions must be judged. These issues are just beginning to be addressed by environmental researchers and require the full attention of both statisticians and atmospheric scientists. Two modes of analysis intermediate to model diagnosis and reliability estimation are sensitivity analysis and uncertainty analysis. The goal of sensitivity analysis is to determine the influence of model input variables or model coefficients on some aspect of the model output. Like model diagnosis, the purpose of sensitivity analysis is to better understand the model, at least under some conditions. Such probing of the model is sometimes performed very roughly by perturbing just one variable at a time to approximately bound the plausible range of ambient concentrations. A more thorough 27 ------- analysis of model sensitivities may require the statistical design of simultaneous perturbations of several variables. Uncertainty analysis is the estimated attribution of the discrepancies between measurements and calculations to, respectively, errors in model input, errors in ambient measurements, model bias, and random variation of the atmosphere for each set of conditions defined by the model input variables. In statistical parlance this is called variance components analysis (if characterization of conditional variability is restricted to conditional means and variances). Uncertainty analysis is relevant to model diagnosis. For example, it can be used to determine whether the apparent poor performance of the model under certain conditions can be plausibly attributed to measurement errors or random atmospheric variation rather than model bias. Uncertainty analysis also pertains to model reliability. That is, the amount of evidence showing that the model "is" or "is not" biased under certain conditions, determines the degree to which such findings generalize beyond the study to other settings or sites. Uncertainty analysis is extremely challenging since the atmosphere rarely provides more than one replicate of the same set of model input conditions. However, the use of either statistical data resampling schemes or wind-tunnel data as a test-bed may yield useful information. In all four types of "model evaluation", it is essential that the evaluation team include or have access to appropriate statistical expertise. 28 ------- APPENDIXD ASRL PROCESS EVALUATION FOR METEOROLOGY DIVISION IN-HOUSE RESEARCH PEER REVIEW ------- ATMOSPHERIC SCIENCES RESEARCH LABORATORY Process Evaluation for Meteorology Division In-House Research Peer Review The Atmospheric Sciences Research Laboratory (ASRL) of the US Environmental Protection Agency convened a panel of scientific experts on September 15 - 18, to review the in-house research of the Meteorology and Assessment Division. The panel consisted of six scientists. These reviewers were asked to evaluate the process involved in preparing and implementing this specific meeting. Five panelists completed the process evaluation. The evaluation instrument was designed to assess the following aspects of the process: 1) Preview Materials; 2) Process and Logistical Information; and 3) the Review Meeting. A section was also provided for reviewers to give their comments and recommendations. The reviewers were instructed to respond to 15 items by circling numbers from 1 to 5 (with 1 representing poor; 2-fair; 3-good; 4-very good; and 5-excellent). Table 1 presents a summary of the reviewers' rating for the 15 items. No items were rated poor. All items received some "excellent" ratings except for overall peer review process and adequacy of time for executive sessions. Most categories were either rated "good" or "excellent". Three items were rated fair: adequacy of time available to preview, meeting purpose, and overall peer review process. Thirteen items were rated excellent: written quality, technical quality, utility for outside reviewer, adequacy of time available to preview, meeting purpose, scheduling of meeting, time/preparation requirement of reviewers, timeliness of meeting notification, timeliness of logistical information, adequacy of time for discussion with EPA staff, quality and utility of presentations, quality and utility of materials disseminated, and support services and activities. 30 ------- TABLE 1. SUMMARY OF PROCESS EVALUATIONS Review Categories Preview Materials 1. Written Quality 2. Technical Duality 3. Utility for Outside Reviewer 4. Adequacy of Time Available To Preview SUB TOTALS Process and Logistical Information 5. Meeting Purpose 6. Scheduling of Meeting 7. Reviewer Responsibilities: Time/Preparation Reauirement 8. Overall Peer Review Process 9. Timeliness of Meeting Notification 10. Timeliness of Logistical Information SUB TOTALS Review Meeting 11. Adequacy of Time for Discussion w/EPA Staff 12. Adequacy of Time for Executive Session 13. Quality and Utility of Presentations 14. Quality and Utility of Materials Disseminated 15. Support Services and Activities SUB TOTALS TOTALS Number of Reviewers Rating Each Item Very Poor Fair Good Good Excellent 1 1 1 1 2 3 1 1 ? 7 1 3 1 1 1 9 3 4 1 1 1 10 21 1 2 1 2 6 1 1 1 3 6 1 1 1 3 15 4 3 3 1 11 1 3 1 4 4 13 2 3 4 3 12 36 31 ------- Is." Comments: § "The organization and support of the peer review by R & E, the Meteorology Division director and his staff is by far the best of all prior ASRL reviews that I have participated in. They are to be commended for their efforts. Reviews of in-house research by a Division are new and require further clarification. The review has elements of personnel action review/appraisal, university tenure review, departmental accreditation, and "traditional" EPA program reviews. It is difficult for peer reviewers to define their task within the existing framework." t "Impossible to examine all preview material: • "Entire process is too brutal." t "Too many people to review with diverse activities." t "The purpose of the review must be carefully considered. My impression is that the review was foisted on ASRL by Washington, DC. To their considerable credit, ASRL management shared with us this perspective on the review. But I feel that the use to the ASRL of such reviews can be better defined and expressed to future panels, after some reflection by the ASRL management." Recommendati ons: • "It would have been nice to have some one-on-one discussion with selected staff." • "Up-to-date vita should be provided." 0 "EPA MD organizational chart would have been useful." t "A time to speak with key people on a one-to-one basis." • "Provide organizational chart to all panel members in advance of meeting. Also, as part of putting the research being reviewed in perspective, it would have been useful to have had a few pages describing the Meteorology Division's mission, long-term objectives, and recent accomplishments - a description at a level of detail somewhere between reference item #1 and individual workplans." 32 ------- APPENDIX E RESULTS OF THE EPA PARTICIPANT SURVEY METEOROLOGY DIVISION ------- Results of the EPA Participant Survey Meteorology Division In-House Research Peer Review The Atmospheric Sciences Research Laboratory of the US Environmental Protection Agency convened a panel of scientific experts to review the in-house research done by the Meteorology and Assessment Division. EPA personnel involved in the program review were surveyed as to the effectiveness and usefulness of the peer review process to them; and to identify problem areas that needed improving. Thirteen surveys were completed. The evaluation instrument was designed to assess the following aspects of the process: 1) review format and logistics; 2) review panel; and 3) your assessment of the peer review process. A section was also provided for comments and suggestions. The participants were instructed to respond to 10 items by circling numbers from 1 to 4 (with 1 representing very satisfied; 2-satisfied; 3-dissatisfied; and 4-very dissatisfied). Table 2 presents a summary of the ratings for these 10 items. None of the participants were "very dissatisfied" with any items, although nearly all items received at least one "dissatisfied" rating. However, the majority of the ratings were in the "satisfied" and "very satisfied" categories. 34 ------- TABLE 2. EPA PARTICIPANT SURVEY SUMMARY Number of Participants Ratinq Each Item REVIEW FORMAT & LOGISTICS 1. Adequacy of time for group discussion with reviewers 2. Adequacy of time for Proiect oresentations 3. Adequacy of time for individual discussion with reviewers SUBTOTALS REVIEW PANEL 4. Reviewers' familiarity with oreview materials 5. Reviewers' demonstrated knowledge of your oroqram 6. Reviewers' expertise in the field 7. Selection of reviewers (Quality of the panel as a reviewina unit) SUBTOTALS YOUR ASSESSMENT OF THE PEER REVIEW PROCESS 8. Support service & activities carried out bv contractor 9. Objectivity & profes- sionalism 10. Effectiveness as a mechanism for OA SUBTOTALS TOTALS Very Satisfied 1 3 2 6 2 2 4 4 12 3 3 6 24 Satisfied 9 7 7 23 10 10 8 8 36 9 9 11 29 88 Dissatisfied 3 3 4 10 1 1 1 1 4 2 2 16 Very Dissatisfied 35 ------- Comments; • "The schedule was overly crowded; it should have been stretched to 5 days, with so many presentations." t "Two of the reviewers seemed familiar with air pollution modeling and meteorology; the others were much less familiar with this sub-field of meteorology. Although they may be professional meteorologists, that is no guarantee they are knowledgable in all aspects of the field. • "This review was unusual in that so many presentations had to be compressed into three days. However, it still seems that most presenters had adequate time for interaction with reviewers as long as they cooperated by allowing time for questions." • "As is usually the case, there was insufficient time allowed for the large number of topics presented." t "Very disappointed in not having an opportunity to set up slides before my presentation because the group did not break for coffee as scheduled." • "I find that the handouts (copies of visual aids) are distracting. The audience begins leafing through the material instead of paying attention to the speaker. I believe the handouts should be done away with." • "Did not believe the process has much benefit for anyone except it provides a defense if attacked. In other words, we can say 'well, the peer reviewers like our program, why are you attacking my budget'. Otherwise, the effort is in essence of no use." 36 ------- APPENDIX F ASRL STAFF RESPONSE TO REVIEWERS' COMMENTS ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY ATMOSPHERIC SCIENCES RESEARCH LABORATORY RESEARCH TRIANGLE PARK NORTH CAROLINA 27711 MEMORANDUM DATE: January 15, 1988 SUBJECT: Response to Peer Review Comments on Meteorology and Assessment Division In-House Research Programs FROM: Francis A. Schiermeier, Director Meteorology and Assessment Division, ASRL TO: Ronald K. Patterson ASRL Peer Review Coordinator The peer reviewers' report on the Meteorology and Assessment Division (MD) in-house research programs is both comprehensive and enlightening. We appreciate their significant efforts, considering that forty-five technical presentations were directed at the reviewers during the three-day schedule. The report contains comments and recommendations on program strengths and weaknesses, on the adequacy of personnel staffing and computer facilities, on maintaining the critical balance between research and applications, and on the Division's role in establishing research priorities. The responses provided here are addressed to critiques, recommendations, and reviewer questions in all of these areas. Responses are not provided for the complimentary statements contained in the report although they have been noted with appreciation. Comments, pp. 3. 17. and Executive Summary; Repeated recommendations on the need to strengthen in-house statistical expertise to support model evaluations. Response: We agree. We are currently recruiting a Ph.D. statistician for the Division but our efforts are hampered by a temporary NOAA hiring freeze. Comment. D. 4; The Meteorology Division seems to be a frequent recipient of requests from other offices and laboratories within the EPA that require redi- rection of program FTE's. The Division appears to be responding to these requests, but at the cost of delaying longer term research needed by the Agency. In some instances...scientific expertise is steered away from model development to application runs. Some of these "quick, turnaround" demands for staff time seem to be appropriate uses of expertise within the MD; however, the balance of short-term and long-term activities should be defined more explicitly. Response: We agree with the ideal of a clearly defined balance between research and applications in the Division but we must also realize that the EPA is a reg- ulatory Agency. As such, it is subject to frequent crises-oriented mandates for redirection of research efforts imposed by Congress and the Administration. 38 ------- Comment, pp. 4-5: One of the major challenges to ORD management would seem to be finding and preserving a balance of staff time allocated to quick response needs of the Agency on the one hand and longer-term needs on the other. Although a good balance seems to have been achieved by the MD, it also seems to the panel that ORD management in general has a responsibility to both the scientific staff and to the Agency to take steps to guard against the erosion of the Agency's research capability... A more active role by the MD is setting research priori- ties would help to guard against the erosion of the MD's research capability by an overload of quick response tasks. Response; ORD management executes a series of steps each year in the budget planning process that includes the ranking of existing research programs and new research initiatives (prepared by Laboratory scientists). These priorities are negotiated by research committees that are composed of both ORD and regulatory program personnel. Unfortunately, this deliberate planning process is then subject to the disruptive redirections described in the previous response. It is thus up to us to sort out these conflicting priorities to achieve the "good balance (that) seems to have been achieved by the MD." Comment, pp. 4-5: More importantly, it should engender a vision and a coherent, internally developed agenda for the MD's research. Even though the HD's tasks are ultimately determined outside the Division, such an agenda would be an important reference for establishing the MD's research priorities. Response: We feel that we do have such an ongoing agenda for the Division's research, and that this agenda is frequently used in sorting out conflicting priorities when they arise. Comment, p. 5: Finally, the emerging program concept of Advanced Research Model Implementation is strongly endorsed by the panel. Response; The establishment of the ASRL Research Modeling Facility is our attempt to prevent further erosion of the Division's research capability by not allowing scientific expertise to be diluted by model applications at the expense of ongoing model development and evaluation. Comment. DP. 7-8; The rotary spectral density application in the analyses of turbulence data is not obviously relevant to EPA objectives even though the quality of the work is good. Such work may not enhance our understanding of boundary layer turbulence significantly. The interaction with the outside com- munity is not very apparent. Response: ¥e take strong exception to the implication of non-relevance. Dif- fusion models, which are the Division's most important product, are basically the application of turbulence theory to practical problems. Since the develop- ment of turbulence theory (and its mathematical treatment) is far from "solved", it is appropriate for us to spend some of our effort in refining the basis for diffusion models. Whether the work will be successful or not is, of course, an open question as it is with a great deal of true research. While it may not have been made apparent to the reviewers, the spectral research has been reviewed very favorably by personnel of the NOAA's Wave Propagation Laboratory. 39 ------- Comment, p. 8; Similarly, the analyses of turbulence data using superposition of autocorrelation functions is not very relevant. The quality of the work is fair and interaction with outside communities not very good. Eulerian autocor- relation functions do not determine diffusion in a turbulent flow. Relating Eulerian autocorrelation functions to Lagrangian autocorrelation functions is not simple and unique. Also "true" Eulerian autocorrelation functions in the atmosphere exhibit oscillation about the zero line thus making the estimate of time scales questionable. Response: Concerning the relationship between the Eulerian and Lagrangian autocorrelation functions, the interpretation of Eulerian autocorrelation func- tions as an indication of the turbulent eddy structure is certainly not new; Sutton (1953) mentions this topic. Although the Eulerian autocorrelation func- tion does not directly determine turbulent diffusion, the literature is replete with attempts to better estimate and interpret the form of this function; some examples are Hisra (1979) and several cited by Moore et al. (1985). Much has been published (for example, Hanna 1981, 1986; Weber et al. 1982) showing the usefulness of estimates of Lagrangian statistics (such as integral time scales) from Eulerian statistics for applications in diffusion modeling. Regarding the contact with the outside community, the autocorrelation study was presented (and received favorable comment) at the AHS Fifth Joint Conference on Applications of Air Pollution Meteorology (Dahai 1986). Comment, pp. 8-9: The Regional Oxidant Model (ROM) appears to be a zero-order model in atmospheric dynamics (at best)... The ROM is a pure diagnostic model, i.e., output concentrations are used (after the fact) to evaluate the effects of alternative strategies for managing air quality planning and standards emission. ROM development apparently needs more computing cycles. Algorithms for the model are not fully understood (for example, incorporation of the effect of clouds and radiative processes). Response: While the reviewers acknowledge the purely diagnostic use of the ROM for assessing emissions control strategies on a regional basis, there seems to be some confusion on the nature of the model itself. There are inferences to the use of better predictive techniques for the atmospheric dynamics portion of the model. The wind fields used in the ROM are derived from observational data and physical principles using a new technique described in Lamb and Hati (1987) that permits explicit treatment of the uncertainty in describing atmospheric motion. This method permits full utilization of dynamical principles in con- structing flow fields for the ROM, but only a portion of the capabilities of the technique have been implemented to date. In addition, flow in the night-time radiation inversion is simulated using a submodel based on solutions of the shallow water equations. Therefore, if the reference to a "zero-order" model pertains to the flow field description used in the ROM, it is incorrect. The wind field specifications are at least "first-order" accurate in their satisfaction of dynamical constraints. In addition, the ROM incorporated schemes for describing vertical mass flux, including cumulus cloud fluxes, horizontal transport, deposition, subgrid scale chemistry effects, etc., that were designed specifically to describe regional scale transport and chemistry phenomena. Unfortunately, there was not enough time in the peer review to describe the numerous facets of the ROM model in any detail. 40 ------- Comment, p. 9: Since a diagnostic model approach is acceptable, more emphasis should be placed on collecting a more representative data set for model develop- ment and performance evaluation. Response; We agree. The regional ozidant program has never been funded suffi- ciently to mount its own field evaluation study. However, as a last resort, we plan to Join in the field study for the Regional Acid Deposition Model (RADH) to acquire some appropriate measurement data for evaluation of the ROM. Comment, p. 9; The ROM model should interact more with numerical weather pre- diction model efforts. Response: Driving the transport portions of air quality models with the output of numerical weather prediction models is a relatively new concept, now being used by a few larger scale models such as the RADM and ADOM for the assessment of acid deposition. This approach is incompatible with the basic premise con- cerning the stochastic nature of atmospheric motion that underlies the method of Lamb and Hati used by the ROM. The integration of four-dimensional data assimi- lation into prognostic wind models is more suited to our needs and it may be explored in the course of implementing our own wind analysis procedure. Comment, p. 9; ...it has been hypothesized that receptor sites can be grouped, such that concentrations averaged over all sites in a given receptor class at a given hour is a quantity that is approximately the same for all realizations of the concentration ensemble. If this is true, then with respect to the receptor class average, the concentration field is a quasi-deterministic variable. There- fore, model simulations of single realizations can be interpreted in the conven- tional deterministic manner. Although this hypothesis has not yet been tested, it is assumed for the present purposes that it is correct. It is unscientific to proceed in this way, though it may be politically and practically sound. Response; The ROM was developed for use in a probabilistic mode. That is, the model output concentrations for a given receptor site should take the form of an expected probability distribution of concentrations, reflecting the stochastic nature of the interpolated flow fields used by the model. The development of the probabilistic flow model has lagged far behind the development on the other components of the ROM system. We therefore now have a fully working model except for this component. Rather than hold up the entire system while awaiting the completion of the probabilistic flow model (which may yet take considerable time to complete), we have chosen to proceed in the quasi-deterministic manner described where observations at aggregates of receptor sites may be matched with corresponding aggregates of model predictions. While it is true that this approach necessitates the use of a-priori assumptions on how the receptors may be grouped, we feel that while imperfect, it is a better route than either put- ting the model on hold while awaiting the final flow model or using the model in a fully deterministic manner which would be completely inappropriate. Comment. D. 9-10; Biogenic emissions inventory development is essential for the application of ROM... The research has involved contact with forest and crop researchers during the development of information sources, but has not resulted in presentation of results to that community. The latter is recommended as a complement to existing reports to the air pollution modeling community. Uncer- tainty estimates were presented but seemed questionable. 41 ------- Response: He agree. A journal article describing the in-house effort on devel- opment of a biogenic hydrocarbon emissions inventory is in progress, although transmittal of results to the forest and crop researchers has been ongoing. He also agree with the reviewers that the uncertainty estimates are questionable. The intention of the presentation was to provide uncertainty estimates for bio- genic emissions using the same methodology currently used by the National Acid Precipitation Assessment Program (NAPAP) for the anthropogenic emissions inven- tory. Thus, at least the relative uncertainties of the biogenic and anthropo- genic emissions could be compared. NAPAP recognizes the weakness of the current methodology but funding for improvements in this area is limited. Comment, p. 10; Work has focused on ozone dry deposition over large areas as determined from aircraft measurements... An effort should be made to complete the analysis of these unique data sets by interpretation of convective struc- tures (for all fields) and treating ozone as a tracer. This has some exciting possibilities. Response; In-house analysis of the aircraft turbulence ozone measurements has indeed been focused on the dry deposition of ozone as noted by the reviewers. The major purpose of the turbulence aircraft flight program during the Northeast Regional Oxidant Study (NEROS) was to obtain experimental data about the spatial variation and temporal behavior of ozone dry deposition over representative land use types. The in-house research to date was intentionally designed to study the variability of ozone fluxes and deposition velocities for different land use areas as functions of time/height in support of the existing in-house regional oxidant model program. There are certainly other interesting aspects of this data set, such as venting of ozone by convective cloud elements and vertical ozone transport over urban areas; these topics have been examined through an extramural effort that was not part of this peer review. Other reviewer- suggested enhancements to the program would be interesting and productive to accomplish, but resources are limited and we must focus on program needs. Comment, D. 11; Feedback between the physicist (s) behind the model development and programming quality control people was a little vague. Scientists that develop the model know the inherent weaknesses. Quality control should gener- ate, independently, a similar set of weaknesses. Response: Interaction between the model developer. Bob Lamb, and the quality control staff is significant even though this aspect was not stressed in the presentation. Any suspect data is immediately brought to the attention of Dr. Lamb who further directs an assessment of the cause and effect of the anomalous data. Summary graphics such as contours of the maximum daily ozone and the QC tracer species are provided to both Dr. Lamb and to Ken Schere, the model evalu- ator, on a routine basis. - P- 11: Hhat about vectorizing the code for the ROM?... Although briefly discussed, the ROM may be improved more by a parallel implementation than by a vector implementation. Response: About one year ago, IBM personnel made a serious attempt to vectorize the ROM code. Benchmark executions showed QO_ improvement in performance of the vectorized code. Subsequent studies performed by the Research Triangle Insti- tute (RTI) explained why ROM is not effectively vectorized, but more likely to 42 ------- benefit from a parallel processing implementation. Therefore, a cooperative research program with RTI is currently implementing a loosely coupled multipro- cessor architecture for the ROM. Comment, p. 11; A more coordinated plan is apparently needed for providing computing facilities to ASRL/MD or even to all of EPA at Research Triangle Park. Also, how much do QC checks slow down the ROM simulation? Response: Since the EPA has acquired an additional IBM 3090 E computer, the computing power of the Agency appears to be sufficient for current and antici- pated ROM applications. ADP timeshare funding for ROM applications is always a limiting factor in the number of ROM executions. However, the requirement for consistent QC checks on the numerous files produced for ROM input is also a limiting factor in the number of new episode periods to be run. Execution and QC of three days of model inputs can require up to a week. This processing involves approximately 60 hours of VAX 785 CPU. Fortunately, the time required to prepare for a control strategy execution is typically less than a day. Comment, p. 11: Also, can EPA use the supercomputer resources at the five national centers? Response: ASRL is also working closely with the EPA National Computer Center to evaluate the need for supercomputer resources within the EPA. A recent study by RTI has concluded that the EPA should acquire shared access or its own (mini) supercomputer at least to explore the potential of the vector supercomputer for some of its future environmental modeling. However, current OMB funding regula- tions restrict the use of research dollars for ADP expenditures. Creative solu- tions may be necessary to use external supercomputer resources. Comment, p. 12: The cloud process module work may evolve into good science... The simplistic approach presented is a reasonable application of the state-of- the-art knowledge for simple cloud model formulation (primarily a thermodynamic emphasis). Effort needs to be extended to properly treat the total entrainment process between the cloud top and the overlying inter facial layer. Response: We agree that the current approach to the cloud process module is simplistic; what was presented was an initial in-house effort. We are currently exploring the use of a non-hydrostatic three-dimensional cloud model for examin- ing some of the most important questions such as cloud top and lateral entrain- ment. Comment. P. 12: The TTNEPH data may be alright for radiation studies but its value in the wet deposition and cloud processing modules is very questionable. Efforts should be made to incorporate the GOES imagery, which is very useful for the ROM domain. Response; He agree. However, the GOES imagery for the 1980 test scenario for the Regional Particulate Model (RPM) were not available in sufficient complete- ness to allow determinations of cloud base and top heights to be made, in addi- tion to the normal areal coverage that most applications require. The RTNEPH data sets already contain this vertical cloud information, as estimated by the Air Force through a best possible method approach. 43 ------- Recent correspondence with the NWS National Environmental Satellite Data and Information Service concerning the GOES NEXT generation of satellites shows promise that, in the coming decade, sufficiently complete data sets will be archived to allow subsequent estimations of vertical cloud definition to be made by our air quality models. Comment. p. 12; The MESOPUFF II modeling-effort is probably important in prin- ciple, but (very good) performance evaluation shows poor performance. The CAP- TEX data set appears adequate to test performance. Best results were obtained for cases of no directional shear. Work should be published in reviewed jour- nals (not just an EPA report). Response: The in-house effort to evaluate and test the MESOPUFF II regional episodic model with the CAPTEX data base consisted of: (1) an operational evalu- ation in which the model was applied with all default features; (2) diagnostic model runs to explore differences with alternative wind fields and dispersion features in the model; and (3) model sensitivity test runs which primarily focused on variations in SOx from exercising model options and changing key parameters in the dry deposition and chemistry modules. The presentation at the peer review showed just a sampling of some of the many statistical evaluation results and graphical displays of observed and model tracer plume paterns. The analysis and interpretation of model results were aimed at identifying reasons for model overprediction and spatial displacements between the respective plume positions. While complete results of this task are being documented in an EPA report, there are plans to publish the notable results in other publications as recommended. Comment, p. 13; In addition to the statistical evaluation, additional effort is needed to outline conditions under which the models are or are not applicable. For example, the simple Lagrangian models show the same performance repeatedly over several years when annual averages are compared. The ISDHE analysis would be more useful if it had led to an understanding of why the models show this type of performance. Response: The goals of the International Sulfur Deposition Model Evaluation (ISDME) focused on the performance of eleven long-term models for each season of one year. These evaluation periods were commensurate with the periods for which the models were designed to simulate the processes. However, to identify the conditions under which these models are or are not applicable, one first must investigate the behavior of the models for periods much shorter than a season, i.e., for a week or less. Although this was once an objective of the ISDME, participating modelers did not submit results for periods considerably less than their interpretation of the appropriate modeling period. Nonetheless, an in- house investigation is currently underway to characterize the behavior of one of these models for daily periods within one season of the ISDME evaluation year. From these results, we should be able to determine those meteorological situa- tions for which the model is or is not applicable. Comment, p. 14: The cloud venting research focuses on the study of penetrative convection, which leads to venting of the planetary boundary layer and transport of pollutants. The results of this vork have important implications for both the ROM and RADM programs and should be continued. 44 ------- Response; We agree. However, NAPAP budget cuts not only prevented further field evaluation studies but even prematurely terminated analysis of field measurement data that had been previously collected. Comment, p. 14: Good, interesting results are obtained in the fluid modeling laboratory and should continue to be published in JFM type journals. The panel, however, questions whether the laboratory experiments are sufficiently reliable simulations of atmospheric flows to be used to set regulatory guidelines. Response; We believe the proper answer to the reviewers' question is, "not if taken in isolation." The results of laboratory experiments have not been used, in and of themselves, to set regulatory guidelines. Many of the studies at the Fluid Modeling Facility (FMF) have been done upon request from the Office of Air Quality Planning and Standards (OAQPS). It is within the purview of the OAQPS to utilize the laboratory results along with many other inputs in setting such guidelines. As a prime example, the dividing-streamline concept arose from FMF laboratory experiments. This concept was validated through the complex terrain field program, then implemented in the Complex Terrain Dispersion Model (CTDH). It is now up to the OAQPS to establish how the results of the CTDM program, including the results of the laboratory experiments, are to be used in the regu- latory process. Comment, pp. 14-15: Inverted tows in stratified cases are adequate, however a deep sheared flow layer is not present (as in the neutral wind tunnel case studies). Experimental fluid mechanics is done well, both in laboratory tech- niques (flow visualization and measurement) and application of theory. How these results are factored into the CTDM was not explained well. Response: The primary value of the work in the stratified towing tank is in providing fundamental understanding of the physical processes involved in stra- tified flows. Much can be learned concerning the structure of stratified flows even in the absence of velocity shear layers. Many aspects of these flows are dominated by the density stratification rather than the velocity shear. Earlier work (not reported to the review panel) showed that under very strongly strati- fied conditions, the shear layer is unimportant. The incorporation of labora- tory results (and complementary field results) into the CTDM was done by the contractor and therefore was not discussed in detail at this in-house review. Comment, p. 15: Stably stratified flows are useful, but do not address the convective planetary boundary layer. Response: The addition of a convective tank at the Fluid Modeling Facility is presently in progress. The most important question to be answered concerning stratified flows in complex terrain is "where is the plume", e.g., impinging on the windward slope, going over the top, or being directed around the side. The question of diffusion about the plume centerline is generally of secondary importance. Comment. D. 15: Although driven by regulatory requirements, the wake effects research (non-aerodynamic structures) can be considered quite basic... Efforts to obtain better flow visualization and, in particular, to quantify those obser- vations are to be commended. Illumination techniques and the statistical evalu- ation of the resulting data need to be improved. 45 ------- Response: He agree. Research on the study of diffusion in building wakes is intended to provide a balance between the immediate needs for regulatory appli- cation and the need to develop a solid research base. Improvements to the illu- mination techniques have been made, and more thorough statistical evaluations of the resulting data are to be pursued once the relation between video image intensity and vertically integrated concentration is firmly established. Comment, pp. 15-16: The work on auto exhaust dispersion is clearly associated with EPA's need to model the impact of a major source of atmospheric pollution. It is difficult therefore to judge what should or should not be done since the work described was very incomplete in itself. Response; The presentation of the results from the study of dispersion in automobile wakes was indeed only a part of a larger project but, as an example of an in-house "wake effects" study, it was included in the peer review. No further work in this area is planned for the near future. Appendix C. pp. 27-28: Comments on Model Evaluation Response: The four categories described by the peer review panel are the major elements pertaining to application of a model to better understand and charac- terize its behavior, performance and capabilities (as distinct from applications of the model to answer a specific policy question). While we might quibble some about the details of the descriptions under each category, possibly seeing them a bit more interconnected than presented by the review panel, we tend to agree. The main point is that a good evaluation requires a team with members who are good at posing informative and useful tests of the model as well as members who are good at quantifying and objectively interpreting those tests, the latter usually being someone with statistical expertise. We believe that our present actions are in concert with the panel recommendation. He have included persons with statistical experience as part of the International Sulfur Deposition Model Evaluation and have statisticians on the model evaluation planning team for the Regional Acid Deposition Model (RADM) evaluation. Reviewer Recommendations. D. 32; Provide organizational chart to all panel mem- bers in advance of meeting. Also... it would have been useful to have had a few pages describing the Meteorology Division's mission... Response: These materials were indeed provided as indicated in the first sentence on page 2 of the peer review report. Reviewee Comment. D. 36; Two of the reviewers seemed familiar with air pollu- tion modeling and meteorology; the others were much less familiar with this sub- field of meteorology. Although they may be professional meteorologists, that is no guarantee they are knowledgable in all aspects of the field. Response: Per my instructions to the ASRL Peer Review Coordinator, the panel was constituted by representatives from the disciplines of statistics, microme- teorology, systems analysis, fluid modeling, and meteorological modeling. This mixture was needed to ensure that at least one reviewer was knowledgable in each of the technical presentations. 46 ------- Comments, DP. 32 and 3fi: Lack of adequate time for presentations. EfiSponge; As noted in some of my preceding responses, the time allotments for presentations were occasionally too limited to accommodate complete descriptions of individual in-house research efforts. For example, the Regional Oxidant Model (ROM) and supporting components were presented in two and one-half hours, where- as the ROM was allotted a full day in an ASRL project-oriented peer review in January 1987. In adhering to this stringent schedule, therefore, presenters were only able to describe the basic tenents of their research projects at the expense of discussing ancillary topics such as relevancy to the Agency's mis- sion, publication of results, and involvement with the scientific community at large. The peer review report shows that, in most cases, the reviewers took this limitation into consideration. References Dahai, X., S. G. Perry, and P. L. Finkelstein, 1986: Multi-Scaling and Exponen- tial Fitting in Autocorrelation Analysis. Extended Abstracts, Fifth Joint Con- ference on Applications of Air Pollution Meteorology, AMS, pp. 348-351. Hanna, S. R., 1981: Lagrangian and Eulerian Time-Scale Relations in the Daytime Boundary Layer. Journal of Applied Meteorology, 20:242-249. Hanna, S. R., 1986: Spectra of Concentration Fluctuations - The Two Time Scales of a Meandering Plume. Atmospheric Environment, 20:1131-1137. Lamb, R. G. and S. K. Hati, 1987: The Representation of Atmospheric Motion in Models of Regional-Scale Air Pollution. Journal of Climate and Applied Meteor- ology, 26:837-846. Misra, P. K., 1979: The Auto-Correlation Function of the Vertical Velocity in the Low-Frequency Range. Preprints, Fourth Symposium on Turbulence, Diffusion, and Air Pollution, AMS, pp. 41-45. Moore, G. E., M. Liu, and L. Shi, 1985: Estimates of Integral Time Scales from a 100-M Meteorological Tower at a Plains Site. Boundary-Layer Meteorology, 31:349-368. Sutton, 0. G., 1953: Hicrometeorology. McGraw-Hill Book Company, New York, NY, 333 pp. Weber, A. H., J. S. Irwin, W. B. Petersen, J. J. Mathis, and J. P. Kahler, 1982: Spectral Scales in the Atmospheric Boundary Layer. Journal of Applied Meteorol- ogy, 21:1622-1632. 47 ------- APPENDIXG THE LABORATORY DIRECTOR'S REVIEW COMMENTS ON THE PANEL REPORT AND THE ASRL STAFF RESPONSE ------- m) UNITED STATES ENVIRONMENTAL PROTECTION AGENCY ATMOSPHERIC SCIENCES RESEARCH LABORATORY RESEARCH TRIANGLE PARK NORTH CAROLINA 2771 1 DATE: February 1, 1988 SUBJECT: Meteorology Division In-house Program FROM: Jack H. Shreffler /V^ H S Deputy Director, AS^L/(MD-59) TO: Ronald K. Patterson TPRO, ASRL (MD-59) I have read the Peer Review report on the in-house research program of the ASRL Meteorology Division and the response by its Director, Francis A. Schiermeier. The report is one of the most complimentary I have re- viewed and the Director's response is complete and adequate in all respects. Moreover, the Summary of Process Evaluations by reviewers indicates a very good preparation and execution of the peer review meeting, for which Research and Evaluation Associates, Inc. and Project Officer Ron Patterson deserve credit. Management should not get involved in fixing things that are not broken. cc: A. Ellison 49 ------- \ § UNITED STATES ENVIRONMENTAL PROTECTION AGENCY ATMOSPHERIC SCIENCES RESEARCH LABORATORY <*/„ ,-0"- RESEARCH TRIANGLE PARK 1 ""& NORTH CAROLINA 2771 1 DATE: February 11, 1988 SUBJECT: Release of Peer Review Panel Report: In-House Research Program By The Meteorology Division FROM: Ronald K. Patterson, Peer Review Coordinator Atmospheric Sciences Research Laboratory (MD-59) TO: H. Matthew Bills, Acting Director Office of Acid Deposition, Environmental Monitoring and Quality Assurance (RD-680) THRU: Alfred H. Ellison, Director Atmospheric Sciences Research Laboratory (MD-59) The enclosed peer review panel report has been reviewed and released by our Laboratory. Comments by ASRL participants and our Laboratory Director are included in the report under Appendices F and G, respectively. We find the comments from our participating staff to be complete and appro- priate. This peer review evaluated only the in-house research program under the ASRL Meteorology Division. A total of 45 technical presentations were made during the four day peer review. The enclosed peer review panel report is highly complimentary of the Division's leadership and the competence of the technical staff. The review panel was impressed by the high degree of excellence exemplified in the areas of experimental fluid dynamics and computational quality control. However, the review panel expressed concern regarding inadequate computer resources and the external pressures exerted on the staff to prematurely release scientific findings. The review panel also discussed the need to strengthen in-house statisti- cal expertise. We are distributing copies of this report and cover letter to the individuals listed below. The ASRL peer review contractor, Research and Evaluation Associates, Inc., has been instructed to forward a copy of this final released version of this report to each peer review panelist. Enclosure cc: Erich Bretthauer (RD-672) Courtney Riordan (RD-682) William Keith (RD-680) Deran Pashayan (RD-680) Morris Altschuler (RD-674) \#4rbert Wiser (ANR-443) Elenora Karicher (RD-680) Jack Shreffler (MD-59) Basil Dimitriades (MD-59) William Wilson (MD-59) Frank Schiermeier (MD-80) Meteorology Division Staff (MD-80) ------- |