ETV Verification Protocol for Urban Runoff Models - Draft 2.0 October 2000 Edited by NSF International Ann Arbor, Michigan 48105 Prepared by Limno-Tech, Inc. Ann Arbor, Michigan 48108 Project Officer Mary Stinson National Risk Management Research Laboratory U.S. Environmental Protection Agency Edison, NJ Office of Research and Development U.S. Environmental Protection Agency Washington, DC ------- TABLE OF CONTENTS TABLE OF CONTENTS i GLOSSARY OF TERMS i ACKNOWLEDGMENTS iii 1.0 Introduction 1-1 1.1 Purpose and Scope 1-1 1.2 The Environmental Technology Verification Program and the Wet Weather Flow Technologies Pilot 1-1 1.3 Verification Testing Process 1-2 2.0 Test Plan Development 2-1 2.1 Purpose of a Test Plan 2-1 2.2 Roles and Responsibilities of Participating Organizations 2-2 2.2.1 Testing Organization (TO) 2-2 2.2.2 Vendor 2-2 2.2.3 NSF International 2-3 2.2.4 EPA 2-3 2.2.5 Technology Panel on Wet Weather Models 2-3 2.3 Model Description 2-4 2.4 Experimental Design 2-5 2.5 Quality Assurance Project Plan (QAPP) 2-5 2.5.1 Quality Assurance Responsibilities 2-5 2.5.2 Data Quality Indicators 2-5 2.5.3 Operational Control Checks 2-6 2.5.4 Data Reduction, Validation, and Reporting 2-6 2.5.5 System Inspections 2-6 3.0 Verification Methods 3-1 3.1 General Elements 3-2 3.1.1 Documentation and User Support 3-2 3.1.2 Error Messages 3-4 3.1.3 Processing Speed 3-4 3.1.4 Pre-processing of Input Data 3-5 3.1.5 Post-processing of Model Output 3-6 3.1.6 Missing Inputs 3-7 3.1.7 Version-to-version Compatibility 3-8 3.1.8 Interface File Capability 3-9 3.2 Hydrology-related Components 3-10 3.2.1 Precipitation Records 3-10 3.2.2 Processing of Precipitation Records 3-12 3.2.3 Water Losses 3-13 3.2.4 Open Channel and Conduit Conveyance 3-15 3.2.5 Pond Storage and Routing 3-17 DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision i ------- 4.0 Data Handling and Reporting 4-1 4.1 Data Management and Analysis 4-1 4.2 Verification Report 4-1 4.3 Verification Statement 4-2 5.0 References 5-1 Appendix A Verification Checklist DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision ii ------- GLOSSARY OF TERMS Accuracy - A measure of the closeness of an individual measurement or the average of a number of measurements to the true value. Includes random error and systematic error. Comparable (favorable) - A qualitative term that expresses agreement between two data sets that were subjected to similar analysis and interpolation. EPA - United States Environmental Protection Agency, its staff or authorized representatives. ETV - The Environmental Technology Verification Program established under the EPA Office of Research and Development to verify the performance of environmental technologies. Hydrograph - A plot of time versus flow at a given location. TO - Testing Organization. GIS - Geographic Information System. Root Mean Square Error (RMSE) - Square root of the Mean of squared difference between two sets of values. Used in the protocol to provide a statistical assessment of the variation between model results and offline calculations. NCDC - National Climatic Data Center. NSF - NSF International, its staff or other authorized representatives. POTWs - Publicly Owned Treatment Works. Precision - A measure of the agreement between replicate measurements of the same property made under similar conditions. Protocol - A written document that clearly states the objectives, goals and scope of verification testing under the ETV Program, the study as well as the test plan(s) for the DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision i ------- conduct of the study. A protocol shall be used for reference during vendor participation in the verification-testing program. QA - Quality Assurance. QAPP - Quality Assurance Project Plan - a written document that describes the implementation of quality assurance and quality control activities during the life cycle of the project. SAG - Stakeholder Advisory Group. Test Plan - A written document that establishes the detailed test procedures for verifying the performance of a specific technology. It also defines the roles of the parties involved in the testing and contains instructions for sample and data collection, sample handling and preservation, and quality assurance and quality control requirements relevant to a given test site. WWF - Wet Weather Flow Technologies Pilot, established under the Environmental Technology Verification Program. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision ii ------- 1.0 Introduction 1.1 Purpose and Scope The purpose of this document is to establish a protocol for the verification of mathematical simulation models used in planning and/or design analysis of water quantity in urban surface systems. The protocol established here will serve as the basis for verification of commercially-available models under the Environmental Technology Verification Program of the United States Environmental Protection Agency (EPA). This protocol describes the steps that ensure that verification is carried out in a consistent and objective manner to assess the relevant performance characteristics of a model. It describes, in general terms, the process of selecting the verifiable tests or documentation evaluations to be conducted, and outlines the methodology to be employed during implementation and documentation. Verification testing, as outlined in this protocol, is intended to confirm the specific capabilities and features of a model as purported in its product literature. The protocol provides guidelines for the preparation of a test plan that specifically addresses a particular model and the testing organization conducting the test. Guidance is also provided on the execution of testing and data reduction, analysis and reporting. 1.2 The Environmental Technology Verification Program and the Wet Weather Flow Technologies Pilot The Environmental Technology Verification (ETV) program is being conducted by the EPA to promote the marketplace acceptance of commercial-ready environmental technologies. The purpose is to provide credible third-party performance assessments of environmental technologies so that users, developers, regulators, and consultants can make informed decisions about such technologies. ETV is not an approval process, but rather provides a quantitative assessment of technology performance measured based on a peer-reviewed protocol. Twelve ETV "Pilots" were established to verify innovative technological solutions covering the range of environmental media. Each Pilot is administered through a cooperative agreement between EPA and a not-for-profit Verification Partner Organization. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 1-1 ------- The Wet Weather Flow Technologies Pilot was established to verify commercially available technologies used in the control and abatement of urban stormwater runoff, combined sewer overflows and sanitary sewer overflows. NSF International (NSF) is the verification partner organization in the Wet Weather Flow (WWF) Technologies Pilot. A Stakeholder Advisory Group (SAG) was formed to assist NSF and EPA in establishing priorities for the verification of wet weather flow technologies. The SAG consists of technology vendors, state and federal regulatory and permitting officials, technology users (POTWs and other municipal government staff), and technology enablers (e.g., consulting firms and universities) with an interest in the assessment and abatement of the impacts of wet weather flows. Since wet commercial weather model software is extensively used by regulators and municipalities to assess wet weather problems and design controls, the SAG recommended that protocol development and testing of these models be pursued under the WWF Pilot. Subsequently, NSF established the Technology Panel on Wet Weather Models to guide the development of a draft protocol for testing commercially available wet weather model software. The Technology Panel includes individuals from industry, academia, engineering and scientific service organizations, and others with expertise in wet weather modeling. NSF contracted with Limno-Tech, Inc. to develop a draft verification protocol. 1.3 Verification Testing Process The process of verification under the ETV Program consists of three primary phases as described below: Planning - The planning phase involves defining the intended capabilities and features of a model and determining the most appropriate means of evaluating these features, culminating in the preparation of a product-specific Test Plan. Guidelines for this phase are described in Section 2 of this Protocol. Verification Testing - This phase includes a start-up period during which the model is installed and input data are developed. The verification testing is conducted once the TO is satisfied that proper installation has been achieved and appropriate input DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 1-2 ------- data are available. Guidelines for this phase are described in Section 3 of this Protocol. Data Assessment and Reporting - This last phase includes all data analysis and the preparation and dissemination of a Verification Report and Verification Statement. Guidelines for this phase are described in Section 4 of this Protocol. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 1-3 ------- 2.0 Test Plan Development 2.1 Purpose of a Test Plan Prior to the start of verification testing of a wet weather model under the ETV Program, the testing organization (TO) shall prepare a detailed Test Plan that clearly describes how and by whom testing is to be conducted. An adequate Test Plan will help to ensure that testing is conducted and that the results are reported in a manner consistent with the requirements specified in this Protocol. A good Test Plan also ensures that information about a particular model is available for incorporation into a Verification Report upon the completion of testing. An individual Test Plan should be developed for each model undergoing verification testing with the understanding that the Test Plans of multiple models prepared by a single TO may have several sections with identical or nearly-identical text. At a minimum, the Test Plan for the verification of a given model shall include: Roles and responsibilities of participants in the verification testing of the model; A description of the site(s) where verification testing is to take place, including the computer system used for the testing; A complete description of the model and its intended functions and capabilities; A description of the experimental design that identifies which tests are to be conducted, the procedures to be followed along with the "challenge tests" being used, and performance measures used; A Quality Management Plan that describes how data quality objectives are to be met; A description of how data is to be analyzed, managed, and reported in a Verification Report. The following subsections provide guidance on the preparation of each required section of a test plan. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-1 ------- 2.2 Roles and Responsibilities of Participating Organizations A Test Plan shall specify the names and addresses of each organization having a role in the verification of a model. Where possible, the Test Plan should include the names, titles, and contact information for specific individuals with designated roles in the verification of the model. Suggested roles and responsibilities for the primary participants are listed below. 2.2.1 Testing Organization (TO) The general responsibilities of the TO include, but are not necessarily limited to: Preparation and review of the Test Plan, including its revision in response to comments from external reviews; Assembly of the necessary technical and support staff to conduct the verification test; Implementation of the Quality Management Plan, as applicable; Communication with verification partner (NSF), EPA pilot personnel, and vendor Coordination and performance of the Verification Tests; Management and analysis of test data and findings; Performance of quality management audits; Development of report findings and conclusions; and Preparation of a draft Verification Report. 2.2.2 Vendor The general responsibilities of the vendor include, but are not necessarily limited to: Review and comment on the draft Test Plan; Review and approval of the final Test Plan; Submittal of a complete, commercially-available model for the duration of the test, along with any ancillary software and supporting documents, such as the User's Manual; Providing descriptive details about the capabilities and intended function of the model as requested by the TO; DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-2 ------- Assisting the TO with the installation, operation, and maintenance of the computer model for the duration of the test, as necessary, including the designation of at least one staff person as the point of contact; Review of the model's Verification Report and Verification Statement. 2.2.3 NSF International As the ETV Verification Partner, NSF International has the following responsibilities: Review and approval of the Test Plan; Overseeing Quality Assurance, including performing technical system and data quality audits, as prescribed in the Quality Management Plan for the Wet Weather Flow Technologies ETV Pilot; Coordinating the Verification Report peer reviews, including reviews by the Stakeholder Advisory Group and Technology Panel; Review and approval of the model's Verification Report; Preparation and distribution of the model's Verification Statement. 2.2.4 EPA EPA personnel involved in the ETV program are responsible for the following: Review and approval of the Test Plan; Review and approval of the Verification Report; Review and approval of the Verification Statement; Posting the Verification Report and Statement on the EPA Website. 2.2.5 Technology Panel on Wet Weather Models The ETV Technology Panel on Wet Weather Models will serve as a technical and professional resource during all phases of the verification of a model, including the review of the generic protocols, model-specific test plans, and Verification Reports. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-3 ------- 2.3 Model Description A Test Plan shall include a complete description of the model and its intended functions, applications, and capabilities. This description shall be prepared as a joint effort between the TO and the model vendor. Statements of performance capabilities and intended applications provide the rationale for distinguishing which among the tests identified in Section 3 of this protocol are applicable to the verification of a particular model. Appendix A can be used as a template for this identification process. Much of the model description contained in the Test Plan will eventually be incorporated into a Verification Report upon the completion of testing. Therefore, the model description should be drafted in a manner that is comprehensible and useful to the prospective model users. The model description in the Test Plan should be consistent with the User's Manual and other product literature distributed by the vendor. Important elements of a Test Plan model description include: Detailed description of the model; Brief history of the model, including predecessor models - both public and private -and the extent of model use to date (number of licenses sold, length of time on the market, etc.); Both minimum and recommended system requirements, if they differ. System requirements should include specification of the computer type, operating system, RAM requirements, hard drive space, graphics packages, and other relevant factors; Specific applications for which the model is intended; Statement of explicit and implicit performance capabilities as indicated in product literature; Identification of any interfacing capabilities with databases and geographic information systems (GIS); Customer support and service mechanisms provided by the vendor or their representatives. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-4 ------- 2.4 Experimental Design A Test Plan shall clearly identify the procedures involved in the verification of a model. Section 3 of this protocol identifies the elements of urban runoff models that are considered, along with guidance on how verification tests and documentation evaluations are to be conducted and reported. The Verification Checklist form, provided in Appendix A of this document, allows the reader to see what tests may apply to a particular model. This checklist shall be completed and incorporated into the Test Plan with guidance from the vendor. One or more of the tests in each category may not be applicable to a given model depending on the intended function or application of the model. If the TO, in conjunction with the vendor, determines that a test is not applicable or cannot be run on a model undergoing verification, the verification report shall document the rationale for its exclusion. The Test Plan shall describe the specific procedures to be employed by the TO in executing the selected tests. These procedures shall be consistent with the guidelines established in the relevant section of Section 3 of this Protocol. 2.5 Quality Assurance Project Plan (QAPP) The Test Plan shall include a QAPP that specifies the procedures that must be used to ensure data quality and integrity. Careful adherence to these procedures will insure that data generated from the verification testing will provide sound analytical results that can serve as the basis for performance verification. This section outlines steps to generate verification data of known quality and sufficient quantity. 2.5.1 Quality Assurance Responsibilities A number of individuals may be responsible for QA/QC throughout the verification testing. However, the TO shall assume primary responsibility to ensure that the simulation tests, the related calculations, and the compilation of data conform to the QA/QC requirements of the Test Plan. 2.5.2 Data Quality Indicators The data obtained during the verification testing must be sound so that conclusions can be drawn on the performance of a model. For all simulation activities conducted for DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-5 ------- performance verification, the NSF and EPA require that data quality parameters be established based on the proposed end use of the data. Data quality parameters include five indicators of data quality: representativeness, accuracy, precision, bias, and completeness. The Test Plan shall include a plan for establishing these indicators. 2.5.3 Operational Control Checks The Test Plan shall describe the QC requirements that apply to the operation of the model and the required computer platform. This section will explain the methods to be used to check on the proper functioning of the system and the frequency with which these quality control checks will be made. 2.5.4 Data Reduction, Validation, and Reporting The Test Plan shall include procedures to maintain good data quality. Specific procedures shall be followed during data reduction, validation, and reporting. 2.5.5 System Inspections On-site system inspections may be conducted as specified by the Test Plan. These inspections will be performed by NSF, or its designated representative, to determine if the Test Plan is being implemented as intended. At a minimum, NSF shall conduct one audit of the TO during the verification of a model. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 2-6 ------- 3.0 Verification Methods This section describes the testing procedures that will be used as guidelines by the Testing Organization (TO) to develop test plans for commercial hydrologic models. It is understood that models vary in their design, intended use, and components. Therefore, the TO is provided with a suite of testing methods from which to select only those tests that apply to the model under study. In addition, this protocol allows flexibility for adding vendor-requested tests for features or components not currently envisioned. All the applicable model elements described in this protocol shall be tested. As new model features or capabilities are added, new tests may be designed and added to the protocol. The tests are separated into the major categories of: General Elements, and Hydrology-related Components. The general elements described in Section 3.1 include items such as documentation, user interfaces, and data processing utilities that are not directly linked to the model's hydrology simulation functions, but which can exert a significant impact on the model's usability. Not all models contain all of these elements. However, where an element exists, it is important to establish the intended function, perform a relevant test, and document the results. The first four general elements (Items 3.1.1 through 3.1.4) do not lend themselves to quantitative testing and analysis. For instance, the available types of model documentation and user support may affect the ease of understanding and usability of the model, but there are many acceptable variations among model vendors for documentation and user support. For these elements, the TO shall elicit all required information from the vendor, perform any evaluations described in the protocol, and document results. The primary performance measures for these elements will relate to claims that are made by the vendor during interviews or correspondence with the TO, or in model documentation. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-1 ------- By contrast, the hydrology-related elements described in Section 3.2 include those functions that directly affect the calculation of storm runoff, water losses, storage, and routing. The goal of testing for any of these elements shall be to ensure that data are properly read into the model and accurately processed to achieve solutions that compare well with unassailable analytical methods. 3.1 General Elements 3.1.1 Documentation and User Support Objective To assess and summarize model documentation and availability of technical support services. Procedure Model Documentation The TO shall obtain from the vendor the available printed and on-line model documentation. Within these two documentation modes, the TO will identify the presence or absence of the following minimum components: "About this software" screen that identifies software name, version number, vendor name, address, and technical support contact information; Identification of computer system requirements, such as hard disk space, single or dual- processor configuration, minimum processor speed, and operating system; Licensing requirements, including description of any hardware or software keys and what the user needs to do to obtain them; Installation instructions; Identification of model limitations, such as maximum number of nodes and/or list of the available parameters; Identification and description of pre- and post-processing options; As appropriate to the model being tested, instructions for required data preparation, pre- processing, model operation, post-processing, sensitivity testing, calibration, and verification; DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-2 ------- Instructions for diagnosing and troubleshooting problems; and Tutorial and/or examples. User Technical Support The TO shall also interview the vendor to determine different methods of documentation delivery and the frequency of documentation updates. Considerations will include: How often is documentation updated? Is documentation updated with each release? How are updates or software patches delivered to existing users who do not upgrade to the latest version? How, and how often, are users informed of new releases, add-ons, patches, and upgrades? What telephone and/or e-mail technical support is provided? Is the software and/or its documentation available for downloading from the vendor's website? Does the vendor maintain an Internet discussion group on its website with, for example, frequently asked questions or frequently encountered error messages? Performance Measure Model documentation will be summarized with any comments on the adequacy of basic components as outlined above. Different types of user support will be documented, including, but not limited to, the frequency of documentation updates, availability of person-to-person technical support, and the method used to notify users of any updates. Data Reporting The TO will document the presence or absence of basic documentation features. Additional user support services, such as fax-back services, searchable technical support databases, and Internet discussion groups, will be noted. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-3 ------- 3.1.2 Error Messages Objective To determine if the error messages encountered by the TO during model testing are documented in the User's Manual. Procedure Error messages encountered by the TO while performing any of the tests described in this protocol shall be documented. The algorithm codes that generate these error messages shall be reviewed. The User's Manual shall be consulted to verify that the messages have been documented and that methods have been suggested by the vendor to overcome them. Any suggested troubleshooting methods will be attempted. Performance Measure The TO's experience relative to error message documentation and troubleshooting recommendations shall be documented. Data Reporting The TO shall record error messages as they appear on the screen, along with presence or absence of associated documentation or guidance. The TO shall document any vendor-supplied trouble-shooting recommendations, and the outcomes of any methods attempted by the TO to overcome the error message. Any undocumented error messages, or failed troubleshooting procedures, will be noted. 3.1.3 Processing Speed Objective To determine and document the model processing speed for simulation of an urban system described in a challenge test. Procedure For urban runoff models, the processing speed shall be tested on a dedicated TO computer running Windows NT or Windows 2000. The TO computer will be a Pentium- class machine with a processing speed of between 500 and 650 MHz. The TO shall identify a challenge test complex enough to allow realistic assessment of the speed required for simulating a continuous dynamic storm situation. For example, a system DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-4 ------- with 200 urban subcatchments would require longer computational time for a 6-month continuous simulation and allow the TO to accurately assess the processing speed. For purposes of the speed test, the TO will set up a simulation that uses the maximum capabilities of the model, up to 200 nodes. At least three runs shall be made to insure representative test results. The processing speed of the model shall be measured in seconds of processing speed per element-month. An example application of this test on the RUNOFF block of SWMM4.4gu can be found in the "mock" test plan and in the "mock" verification report. Performance Measure A hardware and software configuration shall be agreed upon with the vendor. The intent of the speed test is not to compare the relative processing speed of one model versus another, but to provide users with an idea of the typical processing speed that can be expected when a complex system is run with the model. Data Reporting The processing speed, measured from the start of the execution to the end of the model run, shall be recorded internally through a batch program, if possible, or with a stopwatch. The TO will note any unusually long processing events, interruptions, or errors. 3.1.4 Pre-processing of Input Data Objective To determine whether each of the model's built-in input pre-processing capabilities functions as described in the User's Manual. Procedure The TO shall test the input pre-processing capabilities documented in the User's Manual, such as import routines from AutoCAD, standard databases (e.g., MS- ACCESS) and text files (e.g., .CSV files). Each import pathway will be evaluated separately. The typical testing process shall follow the sequence shown below, with the arrow representing the process of converting data from one format to the next format. " Other" software-^ Model Input Model Export Back into "Other" software DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-5 ------- A sample data set containing, for example, 50 hydrological records will be prepared and used for each test. To avoid triggering model interpolation or conversion routines, the data will represent the appropriate time step for the model and will not contain gaps or missing data fields. Specific import errors may include, but are not limited to: Data corruption, including loss of significant digits or rounding errors; Data disordering, resulting from misinterpretation or truncation of data labels; or Model failure resulting from misinterpretation/truncation of data labels or data type mismatch. It shall be noted that this test is limited to verifying the various data import mechanisms available in the model. The accuracy in reading precipitation data and other parameters required by the hydrological processes such as evaporation shall be verified using tests described in Section 3.2. Performance Measure The model import process will be evaluated to determine whether the model has correctly read and processed the data through each of its import pathways. The TO will also determine whether import errors were accompanied by error messages, and whether troubleshooting guidance was available for the error. Data Reporting The TO shall document in narrative form the success or failure encountered during each data import test. 3.1.5 Post-processing of Model Output Objective To assess built-in model output post-processing capabilities such as data export, report generation, and data visualization features. Procedure The TO shall test the output post-processing capabilities documented in the User's Manual. The goal of the testing will be to determine to what extent the processing features accomplish stated goals of the User's Manual documentation. Model output post-processing describes a wide variety of activities, ranging from exporting the model DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-6 ------- results to standard spreadsheet or database software (MS-Excel or Access, etc.) and text files (CSV files, etc.), to generating reports, graphs, and animated visual hydrographic plots. A selection of available post-processing features will be documented and tested. Performance Measure Model post-processing functions shall be evaluated relative to availability of user interfaces, agreement between the operation of the post-processing function and its description in the user documentation, and the plotting options available. Data Reporting The TO shall provide narrative documentation of its experience in implementing the above procedures. 3.1.6 Missing Inputs Objective Models that are capable of data estimation for missing values shall be tested for accuracy. Procedure The TO shall review the model documentation to identify available all data estimation procedures, such as use of default values or use of linear interpolation from adjacent values. If the model uses default values to fill data gaps, model code will be reviewed to determine what values are used and where they are located. If an interpolation algorithm is used, model code will be reviewed to see whether it is written to function according to its description in the User's Manual. The TO shall develop a simple input file containing several data gaps, and use it to test all available data estimation procedures. For comparison, the TO shall also estimate the missing values offline using the same method being tested. Based on these tests, the TO shall answer the following questions: Are the input values used in the interpolation algorithms read in and printed out correctly? For models that use algorithms to estimate missing values, do the algorithms operate as intended, based on the source code and documentation? DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-7 ------- Do the estimated values in the model output compare favorably to those computed using offline calculations? Performance Measure Any differences between modeled values and those computed using offline methods shall be plotted on along 45° line graph, as shown in the graphic below, where offline calculations data plotted on the line, and model results are plotted as they occur in relation to the calculated values. The root mean square error (RMSE) between the model and calculated results shall also be determined to quantify the difference between them. In addition, any variations between the default values or interpolation algorithms in the model code and their description in the User's Manual will be documented. Data Reporting The values from the model and offline methods shall be presented in a database form for direct comparison. Relevant findings from the model code review shall be documented in narrative form. 3.1.7 Version-to-version Compatibility Objective This test evaluates the ability of revised models to reproduce results obtained from previous versions in a consistent manner. 0 ^ a j Calculated Value @ Modeled Value > DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-8 ------- Procedure The TO shall consult the vendor to identify which model versions are currently in use within the modeling community. Within these models, the TO shall identify the algorithms to be tested, which may include those used to simulate water loss, conveyance, storage, and routing, as described in Section 3.2. The TO shall identify relevant tests designed to compare model results with offline calculations, based on tests identified for the corresponding algorithms in Section 3.2. The TO shall then develop simple benchmark input data files to apply to the two most relevant versions, typically the current and previous versions. The TO will also consult model documentation regarding the algorithm enhancements and identify any shift in scientific logic that might lead to different expected results. In the absence of any such change, the model results from both versions will be compared with offline calculations to assess the consistency between versions. Performance Measure The new version shall be evaluated for its ability to replicate results from the previous version, unless the algorithm was revised to produce an expected change resulting from model enhancement(s). Data Reporting Any numerical differences between the results from the two versions shall be documented, along with those obtained from corresponding analytical solutions. Any enhancements between the two versions shall be documented, with differences noted. 3.1.8 Interface File Capability Objective To test the model's capability to create interface files to achieve tasks, such as inputting time series data, and creating links between different model components. Also, any useful features, such as a "hot start" file, shall be tested. Procedure The TO shall develop simple input data files and verify that the interface features operate as documented in the User's Manual. The specific test shall depend on the model components, the functions that are performed, and the selection of available DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-9 ------- monitoring parameters. For example, the TO may input time-stamped rainfall data and then generate the model's binary data file to check the accuracy of the data conversion (i.e., metric to U.S. units) within the model, as well as the organization and structure of the binary file. For testing the use of a hot start file, a model run shall be initiated and then aborted prior to completion. The TO will determine whether a hot start file has been created. The model shall then be configured to use this hot start file, and shall be observed to see if it runs without aborting. If the interface or hot-start files are compiler-dependent (Fortran, C++, etc.), the vendor shall prepare and provide them to the TO according to the specifications of the TO. Performance Measure The TO will determine whether interface features function as described in the User's Manual or not. The TO will further determine whether any other operational assistance features, such as hot start files, function as described in the User's Manual or not. Data Reporting The TO shall document its experience during this test in narrative form. Observations will include listing of interface features found in the documentation, description of the TO's experience locating and using each function, and any errors or difficulties encountered. If additional features are available, such as hot start files, the TO will provide a narrative description of its test. 3.2 Hydrology-related Components 3.2.1 Precipitation Records Objective To test the model's ability to accurately read and/or convert precipitation records provided in a variety of formats and in fixed and variable time steps. Procedure The TO shall evaluate the model's ability to read date-stamped data obtained from standard publicly available data sets; date-independent statistically-developed synthetic design data; and variable time-step data that may be obtained from tipping buckets or DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-10 ------- other publicly available data sources. At the vendor's request, additional data formats will be tested using the same procedure. Date-Stamped, Fixed Time Step Data. The TO shall test the ability of the model to read, as applicable, the following standard publicly-available data formats: National Weather Service, Canadian Atmospheric and Environmental Sciences, Earthlnfo NCDC or Earthlnfo ASCII, and Raw Earthlnfo ASCII. For each of the above formats, the TO shall obtain precipitation records for a one-month period at the finest fixed time step available, not to exceed hourly. The precipitation records will be read into the model and then printed with date and time steps from the model for comparison with actual data. Example input files for the above formats were used to demonstrate the application of this test on the RUNOFF block of SWMM4.4gu. More details of this example test can be seen in the "mock" test plan and the "mock" verification report. Date-Independent Fixed-Time-Step Design Storm Data. The TO shall also test the model's ability to read in statistically-developed synthetic or design storm data, prepared in hourly time steps for a period of one month. The records will be read into the model and then compared with the raw data. Variable Time-Step Data. The TO shall develop separate data sets to test the model's ability to read data at variable time steps, such as would be obtained from a tipping bucket datalogger, or from certain publicly available data sets, such as the National Weather Service. All rainfall records shall be provided in ASCII formatted text, under the headings of: Year Month Day Hour Minute Second Rainfall DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-11 ------- For each data format/time step being tested, the TO shall answer the following questions: Are the precipitation inputs read in and printed out correctly by the model? Do the unit conversions/algorithms described in the User's Manual for processing precipitation values in the format/time step being tested actually exist in the model? If so, do they operate as described? Do precipitation values in the model output compare favorably to those given as input? Performance Measures The TO shall evaluate the model's accuracy in reading and/or performing any unit conversions of the precipitation records in the formats/time steps described above. Data Reporting Precipitation records from the model and those from the original data source shall be presented side by side in a spreadsheet or in a graphical form for direct comparison. 3.2.2 Processing of Precipitation Records Objective To test the model's ability to summarize precipitation records, calculate statistics for data analysis needs, and interpolate data from given precipitation records. Procedure All the summary and statistical parameters described in the model documentation for a specific model shall be included in the testing process. For the purpose of the test, actual or assumed precipitation records shall be prepared, representing time/date- stamped continuous monitoring results from three to four adjacent rain gauges. The data shall describe at least three separate storm events of varying durations and precipitation amounts. Appropriate basin parameters and monitoring station locations required by the algorithms shall be specified in the model. Summary and Statistical Functions. Summary parameters to be tested could include annual or seasonal cumulative precipitation and total number of events, for example. Typical statistical functions include frequency and duration of storms; precipitation depths (individual and cumulative); and average inter-event time. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-12 ------- Weighted Averaging Functions. Many models provide weighted average functions that can be used to interpolate precipitation data from available data points. Weighted average computations will be tested using any of the methods provided by the model, such as inverse distance, closest-point, or the Theissen Polygon method (Wislerand Brater, 1959). For each summary, statistical, or weighted averaging technique tested, the TO shall answer the following questions: Are the precipitation records correctly read into the model, and do they print out properly? Do arithmetic or algorithmic procedures described in the model documentation exist, and do they function as described? Do calculated summary values, statistics or interpolated precipitation values in the model output compare favorably to those computed using offline calculations? Performance Measures Accuracy in summarizing, statistically analyzing, and interpolating precipitation records shall be assessed, using both 45° best-fit line and RMSE. Data Reporting The summaries, statistics, and weighted-average precipitation values computed by the model shall be presented in a spreadsheet side-by-side with corresponding values computed using offline calculations for direct comparison. 3.2.3 Water Losses Objective To ensure accurate implementation of algorithms for characterizing water losses in the urban runoff model. Procedure The TO shall review model documentation and consult with vendors to identify different water loss mechanisms that are included in the model such as: DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-13 ------- Evaporation/evapotranspiration. These hydrologic cycle functions describe the amount of precipitation that evaporates from open water, bare soil, pavement and vegetation. Surface detention. Surface detention variables and algorithms describe the amount and duration of water retained in small depressions, either in impervious or pervious areas. The amount and duration of surface detention are affected by evaporation, evapotranspiration and the basin's infiltration and soil moisture recovery rates. Other soil moisture component variables. These may include soil absorption, deep percolation, water table depth, and infiltration recovery rate. The TO shall review the source code of the model to verify that algorithms have been implemented as described in the User's Manual and in the relevant scientific literature. Based on its review of the model documentation and source code, the TO shall design an appropriate test of each of the model functions. Tests shall be run using a series of 4 to 6 discrete rainfall events and/or a continuous precipitation record in an idealized 100-acre basin. The events shall represent a spectrum of rainfall durations and intensities, such that in half of the cases, precipitation exceeds water losses, and in the other half, it does not. Necessary assumptions shall be made and documented for characterizing each of the water loss components in the basin, including, but not limited to, soil types, land uses, pervious vs. impervious cover, groundwater elevation, etc. The selection of characterization values will be dependent on the specific algorithms used by the model. For example, if Horton's empirical algorithm is used for characterizing infiltration, the TO shall supply the values used for the maximum and minimum infiltration capacities and the infiltration decay. Similarly, if the combination equation (Maidment, 1993) is used for characterizing the evaporation, net radiation exchange for the crop cover, soil heat flux, and vapor pressure deficit shall be documented. Assumed and representative parameter values will be drawn from relevant technical literature. For each precipitation event, an identical set of offline calculations shall be made using the same precipitation data, algorithms, and parameter values. Each test shall be sufficient to allow the TO to answer the following questions about the model capability being tested: DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-14 ------- Are the inputs, such as precipitation records and assumed model parameters, correctly fed in and printed out? Do algorithms described in the User's Manual for characterizing water loss actually exist? Do they function as described? Do the calculated water loss time series data and summary statistics in the model output compare favorably to those computed using offline calculations? Performance Measures The water loss values and statistics computed from the model shall directly correspond with those computed offline. For water losses that are modeled using specific numerical algorithms, the results shall compare favorably to those calculated using appropriate numerical procedures. The results shall be compared using both 45° best-fit line and RMSE. Data Reporting For purposes of comparison and evaluation, the root mean square error from all the data points shall be calculated, and the model results and offline calculation results shall be plotted on a 45° line to see if there is a one-to-one fit. The specific design of the test shall be documented, including all parameters used and any assumptions made. 3.2.4 Open Channel and Conduit Conveyance Objective To determine if the urban runoff model accurately characterizes fluid conveyance through open channels and closed conduits with unsurcharged flow conditions. Procedure The TO shall develop data to define an open channel or closed drainage pipe, with known cross-sectional areas, slope and roughness. A range of constant uniform flows shall be provided as input in the upstream node, and the model will be used to determine the depths and velocities at a selected downstream node. These constant flows shall be assumed as one quarter, one half, and full conveyance capacities of the open channel and closed conduit. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-15 ------- In addition to steady state flows, the model's ability to route unsteady flows will be tested. A hydrograph with a rapid peak and a long recession curve (for example, a flow increase from zero to peak flow at the time of concentration for the basin) shall be given as input to the model, and the attenuated hydrograph at the downstream node shall be computed by the model. An example test hydrograph along with how it was applied to test the RUNOFF block of SWMM4.4gu can be seen in the "mock" test plan and "mock" verification report. Parameters used in the flow simulations shall be assumed and documented based on the algorithm(s) built in the model. For example, for the steady state condition, a representative roughness coefficient is used if Manning's equation (Chow, 1985) has been implemented in the model. For the unsteady flow test, model results for the downstream node will be compared against results obtained using a numerical scheme, such as an implicit finite difference algorithm that solves equations of conservation of mass and momentum. Based on the performance of this test, the TO shall answer the following questions: Are the flow data correctly read into the model, and do they print out properly? Are algorithms for characterizing the channel or conduit conveyances documented in the User's Manual? If so, do they actually exist and operate as described in the model documentation and scientific literature? Do the calculated depths and velocities for different flow values in the model output compare closely to those computed outside the model program? Performance Measures Steady State. The model's ability to generate channel depths and velocities at various capacities shall be evaluated based on a comparison with offline calculations. Unsteady Flow. The TO shall determine whether the model produces results showing the movement of the wave, as reflected in the attenuation of an input hydrograph. In addition, modeled results of an unsteady flow shall be compared to an attenuated hydrograph data obtained through offline calculations. In both steady-state and unsteady flow cases, the comparison between model results and offline calculations shall be made using both 45° best-fit line and RMSE. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-16 ------- Data Reporting Assumptions, parameter values, channel cross-section data, and detailed results for different flow conditions shall be presented. The data report shall also include results computed using unassailable offline methods for direct comparison. 3.2.5 Pond Storage and Routing Objective Verify the accuracy of any built-in algorithms for pond storage and routing. Procedure The routing algorithm provided by the model shall be tested using a simple steady-state flow situation, as shown in the graphic below. 8' Inlet Pipe 30' Storage Pond 50' S' Outlet Orifice The TO shall develop data and parameter inputs needed to model a constant inflow through an eight-foot diameter pipe equivalent to half the total conveyance capacity of the pipe. The inflow shall be routed to a 50' x 20' by 30' storage pond, for example, for 15 minutes, while a two-foot diameter orifice-type outlet continuously drains water from the pond. The TO shall supply assumed values to satisfy the parameters required for testing the pond storage and routing algorithms. For example, if a simple continuity equation is used, the input parameters for the sizes and shapes of the detention pond and its outlet must be documented. The TO shall then calculate flow through the outlet based on the same input data and assumptions using a comparable offline calculation method, such as numerical integration and continuity of storage. Results from the offline calculation shall be compared with the modeled hydrograph. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-17 ------- Based on this test, the TO shall answer the following questions: Are the flow data and physical data of the storage pond correctly read into the model, and do they print out properly? Are algorithms for characterizing the pond storage and routing identified in the User's Manual and the relevant scientific literature? If so, do they actually exist, and do they function as described? Does the calculated hydrograph from the model output compare closely to that computed using offline calculations? Additional test(s) shall be designed by the TO in consultation with the vendor to verify any additional capabilities of the model. For example, if a model is designed to characterize two successive rain events when the pond was only partially dewatered between the events, then the above analytical test shall be modified to reflect partial dewatering. The model results shall then be compared with offline calculations to verify accurate implementation of the model algorithm. Performance Measures The TO shall assess how well the flow from the modeled hydrograph matches that from the offline calculations. Any differences, in terms of hydrograph volume, peak flow over the entire 15 minutes of simulation, and the recession hydrograph shall be evaluated. The differences shall be documented using both 45° best-fit line and RMSE.. Data Reporting Assumptions, parameter values, cross-sectional data, and detailed results from the model for different flow conditions shall be presented. In addition, the hydrograph computed using offline calculations shall be compared with model results and the differences shall be documented. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 3-18 ------- 4.0 Data Handling and Reporting 4.1 Data Management and Analysis A variety of data will be generated during a verification testing. Each piece of data or information identified for collection in the Test Plan shall be provided in the Final Verification Report. The data handling section of the Test Plan shall describe what types of data and information will be collected and managed, and shall also describe how the data will be reported to the NSF for evaluation. 4.2 Verification Report The TO shall prepare a draft Verification Report describing the verification testing that was carried out and the results of that testing. The Verification Report will undergo a complete review by NSF International and the EPA, as well as a peer review as recommended by the Technology Panel on Wet Weather Models. The Model vendor shall review the Report and be provided the opportunity for input on its content. This report should fully describe the model and the verification of its performance characteristics. At a minimum, the Report shall include the following items: Introduction, Executive Summary, Description and Identification of Product Tested, Procedures and Methods Used in Testing, Results and Discussion, Conclusions and Recommendations, References, and Appendices, which may include test data. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 4-1 ------- 4.3 Verification Statement NSF and EPA shall prepare a Verification Statement that briefly summarizes the Verification Report for issuance to the model vendor. The Verification Statement will provide a brief description of the testing conducted and a synopsis of the performance results. The Statement is intended to provide verified vendors a tool by which to promote the strengths and benefits of their product. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 4-2 ------- 5.0 References Chow, V.T., 1985. Open-channel Hydraulics. 21st printing. McGraw-Hill. Maidment, D.R. (ed), 1993. Handbook of Hydrology, McGraw-Hill. Wisler, C.O., and Brater, E.F., 1959. Hydrology. Second Edition, John Wiley & Sons, Inc., New York. DRAFT (10/00) ETV Protocol for Urban Runoff Models, Draft 2.0 - Subject to Review and Revision 5-1 ------- Appendix A Verification Checklist ------- Description of Verification Test Section Number in which details of this Verification Test is presented in the Draft Protocol Applicability of this Test to the Model being Tested (YES/NO) General Elements Documentation and User Support 3.1.1 Error Messages 3.1.2 Processing Speed 3.1.3 Pre-processing of Input Data 3.1.4 Post-processing of Model Output 3.1.5 Missing Inputs 3.1.6 Version-to-version Compatibility 3.1.7 Interface File Capability 3.1.8 Any additional Component(s) appropriate for model being tested 3.1.9+ Hydrology-related Components Precipitation Records 3.2.1 Processing of Precipitation Records 3.2.2 Water Losses 3.2.3 Open Channel and Closed Conduit 3.2.4 Pond Storage and Routing 3.2.5 Any additional Component(s) appropriate for model beina tested 3.2.6+ ------- |