THE ENVIRONMENTAL TECHNOLOGY VERIFICATION PROGRAM^ my ¦ ^UPA Mmd A. W Jfc. ill TtiaiBi I v/trrr ony ' .S, >MlI it>iUliu'hl if I'l^U-iUni! ASMJCY TV, , T U * * ¦ Oak Ridge National Laboratory ETV Joint Verification Statement TECHNOLOGY TYPE: ENVIRONMENTAL DECISION SUPPORT SOFTWARE APPLICATION: VISUALIZATION, SAMPLE OPTIMIZATION, AND COST- BENEFIT ANALYSIS OF ENVIRONMENTAL DATA SETS TECHNOLOGY NAME: Environmental Visualization System Pro (EVS-PRO) COMPANY: C Tech Development Corporation ADDRESS: 16091 Santa Barbara Lane PHONE: 800-NOW-4-EVS Huntington Beach, CA 92649 FAX: (714) 840-2778 WEBSITE: E-MAIL: www.ctech.com evs-info@ctech.com The U.S. Environmental Protection Agency (EPA) has created the Environmental Technology Verification Program (ETV) to facilitate the deployment of innovative or improved environmental technologies through performance verification and dissemination of information. The goal of the ETV Program is to further environmental protection by substantially accelerating the acceptance and use of improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-quality, peer-reviewed data on technology performance to those involved in the design, distribution, financing, permitting, purchase, and use of environmental technologies. ETV works in partnership with recognized standards and testing organizations and stakeholder groups consisting of regulators, buyers, and vendor organizations, with the full participation of individual technology developers. The program evaluates the performance of innovative technologies by developing test plans that are responsive to the needs of stakeholders, conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality assurance protocols to ensure that data of known and adequate quality are generated and that the results are defensible. The Site Characterization and Monitoring Technologies Pilot (SCMT), one of 12 technology areas under ETV, is administered by EPA's National Exposure Research Laboratory (NERL). With the support of the U.S. Department of Energy's (DOE's) Environmental Management program, NERL selected a team from Brookhaven National Laboratory (BNL) and Oak Ridge National Laboratory (ORNL) to perform the verification of environmental decision support software. This verification statement provides a summary of the test results of a demonstration of C Tech Development Corporation's Environmental Visualization System Pro (EVS-PRO) decision support software (DSS) product. DEMONSTRATION DESCRIPTION In September 1998, the performance of five DSS products were evaluated at the New Mexico Engineering Research Institute, located in Albuquerque, New Mexico. In October 1998, a sixth DSS product was tested at EPA-VS-SCM-30 The accompanying notice is an integral part of this verification statement. March 2000 ------- BNL in Upton, New York. Each technology was independently evaluated by comparing its analysis results with measured field data and, in some cases, known analytical solutions to the problem. Depending on the software, each was assessed for its ability to evaluate one or more of the following endpoints of environmental contamination problems: visualization, sample optimization, and cost-benefit analysis. The capabilities of the DSS were evaluated in the following areas: (1) the effectiveness of integrating data and models to produce information that supports the decision, and (2) the information and approach used to support the analysis. Secondary evaluation objectives were to examine DSS for its reliability, resource requirements, range of applicability, and ease of operation. The verification study focused on the developers' analysis of multiple test problems with different levels of complexity. Each developer analyzed a minimum of three test problems. These test problems, generated mostly from actual environmental data from six real remediation sites, were identified as Sites A, B, D, N, S, and T. The use of real data challenged the software systems because of the variability in natural systems. The technical evaluation team performed a complete baseline analysis for each problem. These results, along with the data were used as a baseline for comparison with the DSS results. C Tech Development Corporation staff chose to use EVS-PRO to perform the visualization endpoint on selected data from each of the six sites. In addition, sample optimization was performed for the Site B, N, and S problems, making use of the geostatistical algorithms in EVS-PRO. Cost-benefit analysis (estimates of contaminated volume as a function of cleanup level) was also performed on these three problems and for the Site A cost-benefit problem and the Site D sample optimization problem. EVS-PRO was used to generate several different types of output as appropriate to the problem under study. Output included three-dimensional (3-D) maps of the regions of contamination above specified threshold concentrations as a function of the probability of exceeding the threshold value. A scale of coordinates and surface features were included on the maps to provide a frame of reference. Where aerial photographs were provided, EVS-PRO superimposed the site maps over the photograph to improve visual understanding of the extent of the problem. For the Site A cost-benefit problem, EVS-PRO also generated an animation that provided a 3-D depiction of the extent of contamination. For the Site T groundwater contamination problem, EVS-PRO generated an animation depicting subsurface soil stratrigraphy. These animations rotated the view through 360° to provide the analyst with a more complete view of the data. For Sites B and S, C Tech also provided files generated by EVS-PRO using virtual reality modeling language (VRML) that could be viewed and navigated. Navigation permits the viewer to rotate the image to any angle to gain a better understanding of the extent of contamination. The data from Sites A, B, D, and S were used to generate a cost-benefit analysis of the volume contaminated above the specified contaminant-specific cleanup threshold as a function of probability. For the Site N sample optimization problem, EVS-PRO produced maps of uncertainty as a function of the number of samples collected. This information was used to illustrate the reduction in uncertainty obtained with increased sampling and to highlight regions of high uncertainty that may require further sampling. Several hundred visualizations were produced as part of the demonstration. Details of the demonstration, including an evaluation of the software's performance, may be found in the report entitled Environmental Technology Verification Report: Environmental Decision Support Software— C Tech Development Corporation, Environmental Visualization System Pro (EVS-PRO), EPA/600/R-00/047. TECHNOLOGY DESCRIPTION C Tech's EVS-PRO unites interpolation, geostatistical analysis, and fully 3-D visualization tools into a software system developed to address, among other things, sample optimization and cost-benefit analysis. EVS-PRO's capabilities can be used to provide 3-D maps of geologic structure, subsurface contamination, and regions containing contamination above specified threshold concentrations at a fixed probability level. EVS-PRO can also perform geostatistical analyses that optimize sample locations for site characterization and can estimate volumes and mass of contaminated media for use in cost-benefit analysis. EVS-PRO can quantify the statistical variation in the contaminant volume and mass estimates resulting from the current level of characterization EPA-VS-SCM-30 The accompanying notice is an integral part of this verification statement. March 2000 ------- VERIFICATION OF PERFORMANCE The following performance characteristics of EVS-PRO were observed: Decision Support: EVS-PRO provides decision support through 3-D visualization of environmental data such as contaminant concentration contours, quantification of uncertainties in interpolation predictions, recommendation of additional sample location to reduce uncertainties, and providing statistical information about the extent of contamination (e.g., volume contaminated as a function of probability). Documentation of the EVS-PRO Analysis: For each problem, C Tech provided a detailed description of the steps necessary to import the data into EVS-PRO and perform the desired analysis. The steps proceeded logically, and manipulations to format the data into the EVS format were relatively simple. Numerous files— including visualizations, input files, and output files—were provided for review. Comparison with Baseline Analysis and Data: EVS-PRO produced visualizations from six different sites. All visualizations produced by EVS-PRO were consistent with the baseline data. Visualizations included 3-D representations of geologic structure, hydraulic head, concentration contours above threshold values, and uncertainty maps. The visualizations accurately incorporated surface features (maps of roads, buildings, water bodies) and aerial photographs when available. Visualizations often provided well and sample locations as a function of elevation. Sample locations were accurately color-coded to match the measured data. Sample optimization was performed for Sites B, N, and S. The analyses for Site B and S adequately characterized the plume with an acceptable number of additional samples. For the Site N problem, in which the number of samples was limited, the software inadequately characterized the extent of contamination. EVS-PRO was used to provide cost-benefit analysis of the volume of contamination as a function of threshold concentration and probability level for Sites A, B, N, and S. Its volume estimates were often a poor match to the baseline analysis. EVS-PRO can perform sample optimization analysis to recommend sampling locations and cost-benefit analysis of the volume of contaminated media as a function of probability. To assist the analyst, the software calculates values for the essential parameters used in these analyses based on the data. While the use of these calculated default values makes it easier for the analyst, the values were not always optimal for the sample optimization or cost-benefit analysis. For the Site N sample optimization problem, in particular, approximately a third of the site remained unsampled due to the approach used in EVS-PRO and the limit on the number of samples. For the cost-benefit problems, the estimates of contaminated volumes were often a poor match to the baseline analysis. This was especially true in estimates of volume above the threshold concentration with a low probability of exceeding the threshold. In these situations, the default parameters selected by EVS-PRO often caused predictions of contamination in regions upgradient from the main plume that did not contain data. Operator intervention to optimize model parameters would have led to better, more accurate analyses. The problems identified are a function of the operator and not the software and emphasize the need to have qualified analysts operate the software and for the analyst to examine the model outputs for consistency with the data. Multiple Lines of Reasoning: EV S-PRO provides a number of different approaches to analyzing and visualizing the data, including control over essential modeling parameters. This permits multiple analyses of the data. The software generates statistical and geostatistical information about the extent of contamination, thus providing multiple evaluations to assist in data interpretation. The use of EVS-PRO to generate multiple lines of reasoning assists the analyst in conducting a thorough evaluation of the data. EPA-VS-SCM-30 The accompanying notice is an integral part of this verification statement. March 2000 ------- In addition to performance criteria, the following secondary criteria were evaluated: Ease of Use: EVS-PRO is a sophisticated software product with over 150 computational or visualization modules. The use of visual programming to link the modules makes EVS-PRO fairly easy to use. Most environmental analysts would be able to use the major features of EVS-PRO after two days of training. Advanced features such as use of the scripting language would require more training. An inconvenience of EVS-PRO is its requirement of a fixed-format data field for input files. However, this limitation has been removed in the most recent version of EVS-PRO. Efficiency and Range of Applicability: EVS-PRO efficiently imported, analyzed, and visualized environmental data sets. In the demonstration, the software analyzed four complete problems (three sample optimization/cost-benefit problems and one cost-benefit problem) and two partial problems (perform visualization) with eight person-days of effort. Of these, approximately four days were spent analyzing the data and four days were spent preparing the report. Operator Skill Base: For efficient use of the basic features in EVS-PRO, the operator must have knowledge about contouring environmental data sets and managing database files. To use the advanced geostatistical and statistical features, the operator should be knowledgeable in these areas. Platform: During the demonstration, EVS-PRO was run on a Windows 95 operating system. The computer used for the demonstration was a Pentium II 400 with a Titan II graphics card, 128 MB of RAM, a 4 GB-hard drive, and a 20X CD-ROM. Training and Technical Support: C Tech provides an extensive users' manual documenting code operation and use. Self-paced training modules are available as part of the software package. Technical support is supplied over the Internet and through e-mail. Training courses are available throughout the year. Cost: For a single user EVS-PRO sells for $9995. The EVS pricing structure depends on the product and number of licenses sold to the customer. Discounts are available to educational institutions. Overall Evaluation: The main strengths of EVS-PRO are its outstanding 3-D visualization capabilities and its capability to rapidly process, analyze, and visualize data. The range of visualization output formats and their quality define EVS-PRO as a premier, state-of-the-art visualization system. Its ability to sort and query the data and write scripts to automate repetitive tasks permits EVS-PRO to examine large amounts of data and quickly generate visualizations of the data in many depiction and animation formats. EVS-PRO's object- oriented programming structure allows the many modules to be easily linked together to perform a complex analysis. EVS-PRO is a mature software program that does not have any major limitations. A credible computer analysis of environmental problems requires good data, reliable and appropriate software, adequate conceptualization of the site, and a technically defensible problem analysis. The results of the demonstration showed that the EVS-PRO software can be used to generate reliable and useful analyses for evaluating environmental contamination problems. This is the component of a credible analysis that can be addressed by the software; other components, such as proper conceptualization and use of the code, depend on the analyst's skills. The results of an EVS-PRO analysis can support decision-making. EVS-PRO has been employed in a variety of environmental applications. Although the EVS-PRO software has been demonstrated to have the capability to produce reliable and useful analyses, improper use of the software can cause the results of the analysis to be misleading or inconsistent with the data. As with any complex environmental DSS product, the quality of the output is directly dependent on the skill of the operator. EPA-VS-SCM-30 The accompanying notice is an integral part of this verification statement. March 2000 ------- As with any technology selection, the user must determine if this technology is appropriate for the application and the project data quality objectives. For more information on this and other verified technologies visit the ETV Web site at http://www.epa.gov/etv. Gary J. Foley, Ph.D. Director National Exposure Research Laboratory Office of Research and Development David E. Reichle ORNL Associate Laboratory Director Life Sciences and Environmental Technologies NOTICE: EPA verifications are based on evaluations of technology performance under specific, predetermined criteria and appropriate quality assurance procedures. EPA, ORNL, and BNL make no expressed or implied warranties as to the performance of the technology and do not certify that a technology will always operate as verified. The end user is solely responsible for complying with any and all applicable federal, state, and local requirements. Mention of commercial product names does not imply endorsement. EPA-VS-SCM-30 The accompanying notice is an integral part of this verification statement. March 2000 ------- |