United States Environmental Protection Agency Office of Research and Development National Exposure Research Laboratory Research Abstract Government Performance Results Act (GPRA) Goal 3 Annual Performance Measure 269 Significant Research Findings: Windows/Linux-Based Supercomputing Software for Conducting Uncertainty and Sensitivity Analysis on Environmental Models Scientific Uncertainty analysis (UA) and sensitivity analysis (SA) remain critical, although Problem and often overlooked, steps in the development and evaluation of mathematical Policy Issues models. ORD currently has a significant number of environmental models for which UA/SA still needs to be properly investigated. A fundamental characteristic of UA/SA is the need for high levels of computational capacity to perform relatively redundant model simulations. In juxtaposition, a predominant characteristic of Agency model developers and users is widespread familiarity and reliance upon Windows-based operating systems (OS). While UA/SA is emerging as a critical area for environmental model evaluation, investigation of Windows-based models has been limited by a lack of easy access to supercomputing capacity. Equally, advanced, computation-intensive UA/SA algorithms warrant investigation to determine their efficacy in establishing requisite confidence in their use for regulatory decision-making. UA/SA research supporting evaluation of low and high-order models cannot be accomplished without parallel computing, making possible what would otherwise take hundreds of years using a single personal computer (PC). Supporting both Windows and Linux based modeling systems, an integrated, PC-based supercomputing cluster was recently created at the Ecosystems Research Division (ERD) in Athens, GA. Design of the Supercomputer for Model Uncertainty and Sensitivity Evaluation (SuperMUSE) is described here for the first time along with the software developed to run SuperMUSE. Critical to the success of the approach is this underlying, platform-independent, JAVA-based parallel processing software tool set. The supporting software can be applied in any PC networking environment, and is generally extendable to performing UA/SA on any PC-based computer model. Parallel computing software for Windows/Linux OS is a key element in EPA's ability to investigate and apply a wide range of uncertainty and sensitivity analysis techniques. Success in this area of research is important to our model users in improving quality assurance in their use of simple and complex models to support regulatory decision-making at national, regional, and site-specific scales. ------- Research ERD has undertaken a broad range of programmatic research goals and tasks to Approach improve the "sound science" of UA/SA. PC, Windows-based computational capacity needed to advance UA/SA for low and high order modeling systems is key to this overall research program. In carrying out this work, a PC-based supercomputing cluster, SuperMUSE, was first constructed to facilitate development and testing of a prototype software tool set needed to run this and similar clustered PC systems. Together, the hardware and software tool set approach developed allows one to easily simulate and manage input and output data associated with the many model runs typically needed to conduct UA/SA for an environmental model application. Underlying the parallel computing approach for UA/SA, the distributed program management software set developed here is comprised of three core system-level components. Two of these are model independent and include a server-side CPU Allocator and a client-side Tasker. The Allocator controls overall assignments of simulation experiments on any size cluster, while the Tasker-Client provides TCP/IP-based server-client communication for each PC that registers with an Allocator. The third component, a Model-Tasker, is model-dependent. The Model-Tasker parallelizes a given model's stand-alone PC interface, essentially delivering a list of tasks representing a set of model runs desired. A Model-Tasker registers with a specific CPU Allocator, where several Model-Taskers and Allocators can be active on a given local area network. The prototype itself was developed through UA/SA of a direct national-scale application of the Agency's multimedia modeling system, 3MRA (USEPA, Babendreier, 2003). The software developed here underwent similar rigorous quality assurance testing employed by EPA in the development of the 3MRA modeling system, and was subjected to an extensive peer-review by a 16-member Science Advisory Board panel. In total, the supporting SuperMUSE software tool set was extensively tested through the execution of over 60 million model simulations conducted to date. The software development approach was also subjected to additional internal and external peer-review through submission and acceptance of a conference paper, journal paper, and poster (Babendreier and Castleton, 2002, 2004; Babendreier et al., 2003). Results and The 180+ node SuperMUSE PC cluster was constructed and supporting software Impact Was developed that allows users to more easily distribute and manage parallel execution of "stand-alone" PC models and the many model runs typically needed to conduct uncertainty and sensitivity analyses. As proof-of concept, an initial application was conducted for the high-order, Windows-based modeling system, FRAMES 3MRA Version 1.0. The approach allowed EPA to develop an example uncertainty and sensitivity analysis for a national-scale risk assessment of seven chemicals, involving over 5,400,000 3MRA modeling system runs. With an average "stand-alone" PC model runtime of 160 seconds, SuperMUSE and the software tool set were able to complete the needed modeling system simulations in a period of two months, utilizing 95% of SuperMUSE's equivalent client-PC computing capacity. The methodology, technology, and application developed through this work has already successfully served the Office of Solid Waste (OSW) in a recent, high-profile SAB peer review ------- of the 3MRA Modeling System (SAB, 2004). The outcome of this peer review is anticipated to allow the use of 3MRA to support national-scale decision-making associated with the regulation of land-based treatment and disposal of toxic hazardous wastes throughout the U.S. For 3MRA, comparison of stand-alone PC versus SuperMUSE simulation executions showed a parallel computing overhead of only 0.57 seconds/simulation, a relative cost increase of 0.7% over average model runtime. Parallel computing software tools represent a critical aspect of exploiting the capabilities of such modeling systems. The JAVA software tool set developed and tested on ERD's SuperMUSE cluster readily handled machine and job management tasks in a Windows-OS environment. Compared with other approaches available in the field of supercomputing for environmental modeling, the strength of the SuperMUSE software approach is its ability to support model evaluation tasking for both Windows and Linux PC operating systems. The tool set will ultimately help NERL advance critical research needed for evaluation of Agency models and applications vis-a-vis new guidance under development by the Council on Regulatory Environmental Modeling (USEPA, 2003) and recently adopted EPA guidance on the use of "influential information" (USEPA, 2002). ERD's UA/SA parallel computing research program was carried out through a combination of in-house efforts and key collaborations with two external partners, EPA Office of Solid Waste and the Department of Energy's Pacific Northwest National Laboratory operated by the Battelle Corporation. In addition to the Agency's core 3MRA Modeling Team, other contributors to the design approach and software development work included: Kurt Wolfe and Rajbir Parmar, NERL/ERD, software system development. Examples of recent publications relevant to this study include: Babendreier J.E., Castleton, K. J.. (2004; Accepted). Investigating Uncertainty and Sensitivity in Integrated, Multimedia Environmental Models: Tools for FRAMES-3MRA. Invited paper to appear in Special Edition of International Journal of Environmental Modeling and Software. Babendreier, J.E.., USEPA (2003). The Multimedia, Multipathway, Multireceptor Risk Assessment Modeling System (FRAMES-3MRA Version 1.0) Documentation. Volume IV: Evaluating Uncertainty and Sensitivity. Draft SAB Review Report: EPA530/D/03/001d. Office of Solid Waste and Office of Research and Development, Washington D.C. (see also EPA530/D/03/001a, b, c, e). http://www.epa.gov/ceampubl/mmedia/3mra/index.htm. Babendreier, J.E., Parmar, R.S., Wolfe, K., Uter, S., and McKendrick, M. (2003). PC-based supercomputing for uncertainty and sensitivity analysis of models. EPA Science Forum 2003, Washington, DC, May 5-7, 2003. http://www.epa.gov/athens/forum2003/babendrier i poster.pdf, Babendreier J.E., Castleton, K. J.. (2002). Investigating Uncertainty and Sensitivity in Integrated, Multimedia Environmental Models: Tools for FRAMES-3MRA. In Proc. Of 1st Biennial Meeting of International Environmental Modeling and Software Society, (2) 90-95, Lugano, Switzerland. Research Collaboration and Research Products SAB (2004). Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) Modeling System Panel, http://www.epa.gov/sab/panels/3mramspanel.html ------- USEPA (2002). Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. Office of Environ. Information. EPA/260R-02-008, http://www.epa.gov/qualitv/infonnationguidelines/index.html USEPA (2003). Draft Guidance on the Development, Evaluation, and Application of Regulatory Environmental Models. Office of Research and Development, Office of Science Policy, Council for Regulatory Environmental Modeling (CREM), http://cfpub.epa.gov/crem/cremlib.cfm. Future Research Current work is underway to develop an initial public release of the UA/SA software tool set and documentation for distribution on EPA's Center for Exposure Assessment Modeling (CEAM). Over the next 3 to 5 years, ERD will continue work to create additional customer-based applications along with external collaborations to further transfer and refine this model evaluation technology. Near-future experimentation utilizing the SuperMUSE software resource includes: • Evaluating two promising global-based sensitivity analysis techniques (Regional Sensitivity Analysis and Tree Structured Density Estimation). • Quantifying uncertainty in risk reduction resulting from a national Agency initiative to reduce persistent, bioaccumulative, and toxic (PBT) chemical disposal by 50% by 2005. • Examining UA/SA of an engineered approach to control atmospheric C02 through use of geological systems and underground storage techniques. Questions and inquiries can be directed to: Justin Babendreier, Ph.D., P.E. U.S. EPA, Office of Research and Development National Exposure Research Laboratory Ecosystems Research Division 960 College Station Road, Athens, GA 30605-2700 Phone: 706/355-8344 E-mail: babendreier.justin@epa.gov Federal funding for this research was administered under EPA contract number GS-35F-5338H, Order No. 3D-5386-NBLX. Support for the in-house portion of this project was provided through the U.S. EPA's Office of Research and Development, National Exposure Research Laboratory, Ecosystems Research Division. Contacts for Additional Information ------- |