United States Solid Waste And EPA 510-S-92-801 Environmental Protection Emergency Response May 1988 • Agency 5403W vxEPA Development of Procedures To Assess The Performance Of External Leak Detection Devices Executive Summary - Draft ) Printed on Recycled Paper ------- ------- FINAL REPORT DUE MAY, 1988 DCN 88-231-121-05 DEVELOPMENT OF PROCEDURES TO ASSESS THE PERFORMANCE OF EXTERNAL LEAK DETECTION DEVICES EXECUTIVE SUMMARY DRAFT EPA Contract No. 68-03-3409 Work Assignment No. 2 EPA EMSL Technical Project Monitor: J. R. Worlund Prepared for: CDM Federal Programs Corporation 13135 Lee Jackson Memorial Highway, Suite 200 Fairfax, Virgina 22030 Deputy Project Officer: Dr. M. S. Rosenberg Prepared by: Radian Corporation P. 0. Box 201088 Austin, Texas 78720-1088 Program Manager: T. R. Blair Project Director: A. G. Eklund Principal Investigators: R. D. Achord M. R. Kienbusch D. B. Burrows G. P. Behrens W. L. Crow B. M. Eklund ------- ------- CONTENTS 1. Introduction. 1 Ob j ect ives • 2 Report Organization 3 2. Preliminary Summary Conclusions 4 3. Summary of Technical Approach 6 Vendor Survey & Performance Criteria 6 First Draft Performance Evaluation Test Procedures 6 Device Acquisition 12 Second Draft Performance Evaluation Test Procedures 13 Preliminary Leak Detection Performance Data 13 Validation of Performance Evaluation Test Procedures..... 13 Completion of Proposed Performance Evaluation Test Procedures 14 4. Results 15 Results of Specificity Testing 16 Results of Accuracy and Response Time Testing. 21 References « 35 Appendix A. Mathematical Formulas. 36 Glossary. 47 ------- ------- SECTION 1 INTRODUCTION Underground storage systems comprised of tank, piping, and ancillary components that contain hydrocarbons or other hazardous materials represent a potential source of environmental contamination. Proper system design, in- stallation, operation, and maintenance, along with a leak detection program, can minimize the detrimental effects of leaking underground storage tanks and piping. Devices capable of detecting petroleum hydrocarbons lost from under- ground storage tank (UST) systems can be used inside an UST system, i.e., in-tank, or external to an UST system, i.e., out-of-tank. In-tank (internal) UST leak detectors generally detect losses with liquid-level sensors. Out- of-tank (external) UST detection systems measure the presence of liquid- or vapor-phase hydrocarbons. Early detection of liquid- or vapor-phase petroleum hydrocarbons allows leaking UST systems or components to be removed from service and repaired or replaced, thereby impairment and economic loss of product. service and repaired or replaced, thereby minimizing both environmental Historically, external (out-of-tank) petroleum hydrocarbon leak and release detection devices have not been extensively used and, therefore, are ' used primarily in conjunction with new underground storage tank installations. However, most external leak and release detection systems can be retrofitted at existing facilities. Since existing UST facilities are potentially at greater risk of failure (due to age), it is extremely important that leak and release detection devices be used to detect releases of petroleum hydrocarbons from UST installations. There are numerous commercially available external leak and release detecting devices (for petroleum hydrocarbons) designed exclusively for use with UST systems; however, there are no established performance specifications ------- or certification procedures for assessing the capabilities of these devices. The U.S. Environmental Protection Agency (EPA) is currently promulgating regulations and guidelines that address underground storage tanks. One aspect of this program is to develop simple, benchmark test procedures that can be used to assess the performance of external petroleum hydrocarbon leak and release detection devices. Performance test results for external leak and release detectors in actual field use may vary from performance measured by these benchmark tests. Radian Corporation, as a subcontractor to Camp Dresser McKee under con- tract to EPA/EMSL, is providing technical support in developing test proce- dures for performance evaluation of out-of-tank (external) leak or release detection equipment (components) for petroleum hydrocarbons. Several tasks critical to this program are complete or currently underway. Results of a survey of vendors of external petroleum leak monitoring devices for use with underground storage tanks have already been published (1). Standard test procedures are under development to allow objective comparisons of external leak and release detector performance. Draft performance methods designed to be simple benchmarks for comparison of external leak and release detectors under prescribed conditions have been submitted to EPA (2). OBJECTIVES The overall objective of Radian's UST work is to support EPA's regulatory development. Radian's key contributions are to identify parameters that affect leak detector performance and to develop standard test procedures for assessing external petroleum hydrocarbon detector performance within the meaningful ranges for each performance parameter. The objective of the cur- rent task (Work Assignment 02) was to develop and execute draft standard test procedures for performance evaluation of external detection systems for leak and release monitoring. Procedures that address specificity, accuracy, and response time for both vapor-phase and liquid-phase external detectors have been submitted to EPA in a draft report (2). ------- The performance evaluation test procedures submitted to EPA for review were written as a laboratory test plan for Radian personnel involved in devel- oping formal procedures. Each procedure will require validation before it is used by the public. These performance evaluation procedures will also require revision to provide further clarification, example calculations, and an increased level of detail for performance by an operator. A final collection of performance evaluation test procedures for external leak and release detec- tion components will be submitted in a single document at a future date for EPA review and validation. REPORT ORGANIZATION Preliminary conclusions and a summary of results from initial execution of the performance -evaluation test procedures for a limited number of external leak and release detection devices are given in Section 2. Section 3 provides a description of the technical approach for this work assignment in the con- text of prior work. Section A gives a preliminary discussion of the results from initial testing of external leak and release detection equipment using the test procedures for evaluating performance. References, mathematical formulas used for vapor-phase and liquid-phase detector performance calcu- lations (Appendix A), and a glossary are given at the end of this summary report. ------- SECTION 2 PRELIMINARY SUMMARY CONCLUSIONS Test procedures used to date to assess the performance of external petroleum hydrocarbon leak and release detection systems were developed to provide a uniform, inexpensive, benchmark protocol for comparing parameters such as specificity, accuracy, bias, precision, and detection time. Although only 13 detectors have been tested by Radian, the procedures appear to be applicable to a wide variety of external petroleum hydrocarbon detectors. Categorization of external detectors provided adequate separation of operating principles for ease of performance testing. Intermittent and continuous detectors, separated into liquid-phase and vapor-phase categories, were easily tested with the same procedures. No major problems or obstacles were reported by Radian personnel per- forming these initial test runs, and it is likely that these procedures could be performed easily by manufacturers or technicians knowledgeable in UST technology. Preliminary test results of the performance evaluation procedures indi- cate that external vapor-phase and liquid-phase detectors respond to most petroleum hydrocarbon components; however, specific detection systems (espe- cially vapor-phase sensors) may have better response factors for certain com- pound classes (aromatics or aliphatics). These results also indicate that the 250 ppmv concentration used for specificity testing of vapor-phase detectors may be too low for evaluation of some types of qualitative-output detectors. All the liquid-phase detectors responded within 24 hours or less to a 0.5-inch thickness of petroleum product layer floating on water. ------- Liquid-phase accuracy, bias, precision, and response time results appear to be independent of the composition of the hydrocarbon test liquid; there- fore, the simulated standard gasoline mixture probably provides a good repre- sentation of a commercial product while offering the advantages of a uniform, laboratory-generated test liquid. In general, the liquid-phase detectors that were tested responded in the presence of a hydrocarbon. A larger range of accuracy was observed for the vapor-phase detectors that were tested. Detection times for vapor-phase detectors ranged from less than 15 seconds to 24 minutes, with most of the responses occurring in less than 1 minute. Vapor-phase fall time values were all under 15 minutes, and most values were under 1 minute. Detection times for liquid-phase detectors were generally less than 2 hours and were less than 15 minutes for most detectors when the hydrocarbon layer thickness was above the vendor's specifications for lower detection limit. Future testing of a wider array of external leak and release detection devices may require modification to test containers and chambers to accommo- date all detector probe configurations. New performance evaluation procedures may be required to address the scientific principles of operation of future detection equipment as technology presents new modes of external leak and release monitoring. For test procedures to progress past the draft stage, validation tests will be required to evaluate each procedure's acceptability, and possibly, the relationship of leak detector performance in the laboratory to leak detector performance in the field. ------- SECTION 3 SUMMARY OF TECHNICAL APPROACH VENDOR SURVEY AND PERFORMANCE CRITERIA (WORK ASSIGNMENTS 44 AND 70) The external petroleum hydrocarbon leak and release detection devices that were identified in a prior study (1) were categorized by function into four groups (i.e., intermittent and continuous devices for both vapor-phase and liquid-phase petroleum hydrocarbon detection). Separate performance eval- uation test procedures are being developed for vapor-phase and liquid-phase petroleum hydrocarbon detection systems, but intermittent and continuous detection systems of each type are being tested using identical procedures. Parameters that describe petroleum hydrocarbon leak and release detection device performance have been identified (3) and are being used to develop performance evaluation test methods. Parameters related to long-term perfor- mance (reliability), safety, and cost were not considered in the development of the present test procedures. FIRST DRAFT PERFORMANCE EVALUATION TEST PROCEDURES (WORK ASSIGNMENT 92) In March 1987. a first draft of example performance evaluation procedures was submitted to EPA for review (4). These procedures addressed: 1) speci- ficity of vapor-phase leak and release detection equipment and 2) accuracy and response time for liquid-phase detectors. These test procedures were based upon information from the literature, vendor and manufacturer testing, and engineering and scientific knowledge of the equipment to be tested. Prior to writing the first draft, performance parameters were grouped together, where possible, to save time and resources and to decrease the ------- variability of reproduced test procedure performances. The performance param- eters were combined in some cases, as shown in Table 1. so that initially a total of 10 performance evaluation test procedures were to be written. Proce- dures have been written in EPA's standard format. The generic performance procedure outline is given in Table 2. The following paragraphs describe each of the outlined sections' objectives. The Scope and Application section of each procedure names and describes the performance parameter(s) that the procedure was designed to measure, such as accuracy, signal-to-noise ratio, or specificity. The applicability and limitations of the procedure are discussed. Procedures are specific to either vapor-phase or liquid-phase detectors but are usually applicable to both intermittent and continuous external leak and release detectors. Technical limitations of the method are given. The Stunmary section provides a brief overview of operational concepts and principles that are necessary for performing the procedure. The Interferences section is a detailed guide on how to detect and mini- mize potential interferences. Specific recommendations on how to measure the effects of contamination are made in this section. The Safety section provides information and guidelines that are needed to perform a procedure safely. This section discusses minimum safety practices that must be implemented to reduce the possibilities of an accident or exposure. The Apparatus and Materials section describes equipment, materials, and specifications for each. Descriptions of test containers or chambers, gas rotameters, manometers, and timers are included in this section. Materials such as tubing and fittings are covered also. ------- TABLE 1 PROPOSED PERFORMANCE EVALUATION TEST PROCEDURE MATRIX FOR EXTERNAL LEAK AND RELEASE DETECTION EQUIPMENT 1) Test Procedure for Measuring Accuracy and Response Time for Liquid-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 2) Test Procedure for Measuring Accuracy and Response Time for Vapor-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 3) Test Procedure for Determining the Specificity of Liquid-Phase Out-of- Tank Petroleum Hydrocarbon Leak and Release Detectors 4) Test Procedure for Determining the Specificity of Vapor-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 5) Test Procedure for Determining the Detection Limit and the Signal-to- Noise Ratio of Liquid-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 6) Test Procedure for Determining the Detection Limit and the Signal-to- Noise Ratio of Vapor-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 7) Test Procedure for Measuring Drift for Liquid-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 8) Test Procedure for Measuring Drift for Vapor-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 9) Test Procedure for Determining the Effect of Potential Interferences of Liquid-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors 10) Test Procedure for Determining the Effect of Potential Interferences of Vapor-Phase Out-of-Tank Petroleum Hydrocarbon Leak and Release Detectors ^Submitted to EPA in draft form for review in March 1988. ------- TABLE 2. OUTLINE FOR PERFORMANCE EVALUATION TEST PROCEDURES FOR--EXTERNAL LEAK AND RELEASE DETECTION EQUIPMENT 1. Scope and Application Method Covers (e.g.. detection limit) Application (e.g., continuous gas-phase detectors) Limitations Experienced Personnel to Employ Method 2. Summary of Method Brief Discussion of Method Principles Optional Method(s) : 3. Interferences Sources of Contamination Practices/Materials to Avoid QC Checks for Contamination 4. Safety Compliance to OSHA Regulations Special Chemical Concerns Safety Procedures 5. Apparatus and Materials Major Systems Testing equipment Functioning ranges Specifications Alternative systems V- 6. Test Gases or Liquids Chemicals/Gases Define if necessary Purchasing information Procedures to make solutions Storage requirements Expiration QC reagents discussed in QC section 7. Calibration Per Instrument Used Stepwise procedures Alternative Procedures (include equations/define variables) (Continued) ------- TABLE 2. (Continued) 8. Quality Control Established QC Program Minimum requirements Record keeping Special procedures Accuracy and precision Blanks Spikes Ongoing system control 9. Procedure •• . Stepwise Guide Through Method, Including: Operating conditions Estimated responses Calibration QC Atypical procedures 10. Calculations Discussion of All Calculations Definition All Variables Used 11. Method Performance (i.e., how reliable is this method) Reliability of Method Specific ranges (if applicable) Operator Overall Method Accuracy 12. Definitions For terms that have meanings that are particular to the method or are ambiguous 13. References 10 ------- The Reagent section describes requirements for all chemicals that are used in a procedure. This section is entitled "Test Gases" or "Test Liquids" for vapor-phase or liquid-phase methods, respectively. Requirements include concentrations, tolerances, and quality or purity. Storage recommendations and necessary mixing instructions are also given. The Calibration section provides a stepwise guide for calibrating equip- ment. This equipment may include rotameters, timers, thermocouples, etc. Frequency and acceptance criteria for calibrations are provided. Calibration of actual petroleum hydrocarbon detectors is only briefly discussed in this section. Complete calibration instructions for detectors should be provided by the manufacturer or vendor. The Quality Control section describes minimum steps and acceptance cri- teria that are necessary to establish that procedure results are reliable. Quality control steps may include replicate analyses, record keeping, and blank testing. Frequency, acceptance criteria, and corrective actions for quality control methods are listed in this section. The Procedure section includes detailed step-by-step instructions of how to perform the procedure. This section guides the technician through equip- ment assembly, quality control procedures, calibration, testing, and expected results. The Calculations section provides all the equations necessary to perform the procedure and reduce resulting data. Equations for quality control, data manipulation, and data reporting are provided. The Method Performance section gives the technician an impression of the procedure's reliability. This section addresses procedure accuracy, repeata- bility, reproducibility, range, applicability, ruggedness, testing duration, and limitations. (This section can be completed only after validation testing 11 ------- and is incomplete in the methods that have thus far been submitted for review.) The Definitions section is a list of terms and their meanings that apply specifically to these methods. References provide a list of references that were used in writing the procedure. DEVICE ACQUISITION (WORK ASSIGNMENT 02) After sets of representative vapor-phase and liquid-phase detection devices were identified and categorized, suppliers were contacted. In August 1987. Radian Corporation began acquiring a limited number of external leak and release detection devices from vendors to undergo initial execution of newly drafted performance evaluation test procedures. Efforts were made to acquire several devices from each category of sensor type established in the prior vendor survey (1). Several selection criteria were established to guide selection of the external leak and release detection devices to undergo initial performance testing. The selection criteria were: detector type, number of devices in service, specificity for UST leak detection market, and commercial availabil- ity. Vendors were contacted; agreements with Radian to borrow the sensor systems for testing were executed; and the devices were sent to the Radian laboratories for performance evaluation testing. A more detailed description of the device acquisition plan was presented in a draft technical note to EPA (5) and subsequently updated in monthly progress reports. Devices that were acquired represented several categories of scientific operating principles within the larger categories of liquid-phase and vapor- phase leak and release detection devices. The following types of external petroleum hydrocarbon liquid-phase leak and release detection devices were 12 ------- acquired: electrical conductivity sensors, interface probes, product- permeable probes, and product-soluble sensors. External petroleum hydrocarbon vapor-phase leak and release detection devices were representative of the following sensor classes: metal oxide semiconductors, diffusion sensors, catalytic sensors, photoionization sensors, and infrared sensors. SECOND DRAFT PERFORMANCE EVALUATION TEST PROCEDURES (WORK ASSIGNMENT 02) After the draft performance evaluation test procedures were executed on a limited basis in the Radiap laboratories, these procedures were modified from the procedures given in the Quality Assurance Project Plan for Work Assignment 02 (6). The second draft performance evaluation test procedures were sub- mitted to EPA in March 1988 (2). Modifications included minor changes to the test apparatuses to accommodate different sensor shapes and sizes. PRELIMINARY LEAK DETECTION PERFORMANCE DATA Only 13 external petroleum hydrocarbon leak and release detectors were tested using the draft procedures. No more than two detectors representing any one operating principle were tested. Therefore, the discussion of results (Section 4) of this initial phase of testing using draft procedures is limited. No specific conclusions have been made about the adequacy of opera- ting principles from performance of the draft test procedures for performance evaluation. Since these methods have not been validated, these data should be treated as preliminary. VALIDATION OF PERFORMANCE EVALUATION TEST PROCEDURES •Validation tests will be developed and performed in the next phase of this program. Individual tests for validating each of the draft performance evaluation test procedures will be developed. The major objective of these validation test procedures will be to evaluate the ability of each of the draft procedures to produce valid test data within acceptable error limits. 13 ------- The results from performing the validation tests (in a future work assignment) will be used to identify any limitations in the performance evaluation test procedures. The exact details of each validation test will depend on the associated performance test procedure. The overall approach to be used in developing the validation process will be the same for all cases and is patterned after the successful approach developed by the EPA for the new source performance test methods. The overall approach consists of the following steps: • Technical examination; • Laboratory evaluation; • Ruggedness testing; and • Validation testing. Subsequent to the validation testing, final versions of performance eval- uation test procedures will be submitted for EPA review. Final validation may require establishing the relationship of laboratory performance to field per- formance for network leak detection design. COMPLETION OF PROPOSED PERFORMANCE EVALUATION TEST PROCEDURES The purpose of developing performance evaluation test procedures is to provide a means of establishing a uniform, consistent database of performance characteristics for external leak and release detection and monitoring devices. Over the long-term course of this project, it is likely that other procedures for new technologies and additional performance parameters to those given in Table 1 will be identified. As performance evaluation procedures are refined-and validated, the procedures document will be updated. 14 ------- SECTION 4 RESULTS This section is a summary of results and observations made during the preliminary execution of performance evaluation test procedures to determine specificity, accuracy, bias, precision, and response time of external petroleum hydrocarbon leak and release detection systems. The test matrix given in the project plan (6) and procedures given in the draft methods manual (2) were followed. None of the results presented in this section have been validated or approved for disclosure by EPA or any of its contractors. The performance evaluation results obtained for specific devices were secondary to the task of developing test procedures. Although all due diligence was taken in the exe- cution of these procedures, the results summarized below must be considered preliminary for several reasons. First, the draft procedures performed have not yet been scientifically validated. Changes in the procedures may alter the responses observed for some of the devices. Secondly, the number of devices tested was extremely small compared to the universe of devices on the market. Therefore, no correlation trends assigned to specific operating prin- ciples can be made at this time. Although a wide variety of operating princi- ples, were represented, no more than two detectors from a single category were tested. Again, the major emphasis of this work was to develop test proce- dures, not to test the devices. The following discussion is organized by the performance categories: specificity, accuracy, and response time. Within each performance category, the test matrix, test procedure, and results are organized under subcategories for vapor-phase and liquid-phase detectors. Special efforts, have been made in 15 ------- the data presentation to maintain the anonymity of devices and associated principles of operations, as agreed upon by the participating vendors. Radian Corporation, and EPA Environmental Monitoring Services Laboratory in Las Vegas, Nevada. RESULTS OF SPECIFICITY TESTING The significance of specificity (degree of response) of an external leak detector is twofold. If a detector's response is not specific, it may respond to compounds in the environment that are not associated with UST petroleum hydrocarbon releases, causing a false alarm (false positive) situation. On the other hand, if the detector is highly specific (selective), then a low probability of detection may result (false negative). Therefore, one of the most important performance evaluation parameters for external petroleum hydro- carbon leak detection systems is specificity. Specificity answers the ques- tion, "What will the detector detect?" Specificity is especially important when the stored product is a single-component hydrocarbon. Proper network design, site-specific factors, and calibration for exter- nal detectors will affect field specificity. These issues were not addressed in these laboratory benchmark test procedures. Vapor-Phase Detectors - Specificity Response The specificity of a vapor-phase detector was defined for the procedure as a system's ability to indicate either the presence or concentration of a pure chemical or a mixture of petroleum hydrocarbons at 250 ppmv (certified gas concentration in ultrahigh-purity air). The test gases were monocomponent n-hexane, benzene, toluene, n-butane, isobutane, 2-methylpentane. and 3- methylpentane gas standards. Each of the seven vapor-phase leak detection systems was randomly tested with each gas near 250 ppmv with 20% replicate measurements. The replicate quality control test gas was chosen at random for 16 ------- each leak detector. Detector responses were initially measured in ultrahigh- purity air, monitored for response to a test gas, then purged with ultrahigh- purity air according to the test matrix given in the project test plan (6) using the second draft test procedure (2). Results from these tests were used to calculate response factors for quantitative-output detectors for each test gas, and activated versus nonactivated responses for qualitative-output detec- tors for each test gas. These results are summarized in Table 3. Three qualitative-output vapor-phase detectors (A, B, and C) failed to respond to any of the test .gases at a concentration of 250 ppmv. These detec- tors were subsequently tested with n-hexane at 500 ppmv, and only Detector C responded at the higher concentration. These detectors had similar modes of operation and do not represent the wide variety of qualitative-output vapor- phase detectors on the market. The four quantitative-output vapor-phase detectors (D, E, F, and G) showed specificity for the test gases, with differences in response factors. Some detectors were specific to the chemical class of compound (aliphatic versus aromatic), while others showed specificity response factors that appeared to be a combination of chemical class and other factors such as molecular weight. For example, Detector D was most specific (had high response factors) for 2-methylpentane, isobutane, n-hexane, and isopentane, respectively. It was less specific (had lower response factors) for n-butane and toluene. While Detector E's specificity was consistent, response factors exceeded 100% for four of the test gases. The gas concentrations tested were at the lower end of Detector E's operational range. Detector F was most spe- cific for benzene and toluene (aromatic compounds) but demonstrated very low specificity response factors for the other test gases consisting of aliphatic compounds. Detector G had accurate specificity for n-hexane and 2-methyl- pentane but lower response factors for the other aliphatic compounds (n- butane, isobutane, and isopentane) and very low response factors for benzene and toluene (common aromatic components in fuels). 17 ------- oo TABLE 3. SUMMARY OF VAPOR-PHASE LEAK DETECTORS SPECIFICITY TEST RESULTS8 (Percent Response to Test Gases) Bact Detector Identification Ul :ground n- 1P Air Butane Isobutane Test Gases n- 2-Methyl- Isopentane Hexane pentane Benzene Toluene Qualitative-Output Detectors A B C Quantitative-Output D E F G NA NA NA I NA NA NA NA NA NA . Detectors NRd 11 63 NRd 100 200 NRd 1 1 NRd 68 63 NA — b NA . •» NA ~b NA NA ~° NA 57 61 65 200 200 200 6 10 12 77 96 92 .. NA NA NA NA NA NA 42 17 100 150 100 102 2 14 Specificity, the ability of a detector to indicate the presence or concentration of a test gas at 250 ppmv. is reported in terms of response factors calculated as the percent of the detection system's response compared to the certified concentration of the test gas for quantitative-output detectors and as "activated" (A) or "non-activated" (NA) for qualitative-output detectors. bNo response to either 250 ppmv or 500 ppmv of n-hexane. CActivated response to 500 ppmv of n-hexane but no response to 250 ppmv of n-hexane. dNR = no response for quantitative-output detectors. ------- Liquid-Phase Detectors - Specificity Response For liquid-phase petroleum hydrocarbon leak and release detection sys- tems, specificity was defined in the performance evaluation test procedure as the response to a 1.27-cm (1/2-in.) layer on water of different chemical substances representing components or types of petroleum hydrocarbons commonly stored in UST systems. The test liquids used in the performance evaluation procedure were: n-hexane, xylene(s), toluene, commercial gasoline, a laboratory-mixed "standard" gasoline, commercial diesel fuel, and commercial jet fuel. Each of six external liquid-phase petroleum hydrocarbon leak and release detection devices was randomly tested using each test liquid with 20% replicate measurements. The replicate quality control test liquids were chosen at random for each liquid-phase detector. Detectors were measured for background responses in tap water, then for responses in each test liquid with a complete cleaning of the test apparatus between tests. The matrix from the project test plan (6) and the second draft test procedure (2) were followed. Results from these tests were used to determine if there was a response for specific compounds within 24 hours. Test results indicated five of the six liquid-phase detectors responded to a 1.27-cm (1/2-in.) thick layer for all seven test liquids on water within 24 hours. The sixth liquid-phase detector responded to all test liquids within 24 hours, except commercial diesel fuel, to which it responded in 26 hours. The ranges of observed response times during specificity testing are summarized by test liquid in Table 4. The longest time until response for all test liquids was observed for the same detector (Detector D). Likewise, the shortest time required until response to all test liquids was observed for a single detector (Detector B). Detector D required over 16 hours to respond to n-hexane, xylene(s), diesel fuel, and jet fuel; it responded to toluene, commercial gasoline, and synthetic gasoline mixture in under 2 hours. Detector B showed no specificity 19 ------- TABLE 4. SUMMARY OF RESPONSE TIMES TO TEST LIQUIDS DURING LIQUID-PHASE SPECIFICITY TESTING Test Liquid , Ranges of Observed^ Time Until Response' n-Hexane Xylene(s) Toluene Simulated Gasoline- Commercial Gasoline Diesel Fuel Jet Fuel <1 sec - 17.5 hr <1 sec - 21.5 hr <1 sec - 1.5 hr <1 sec - 1 hr <1 sec - 1 hr <1 sec - 26 hr <1 sec - 16.5 hr aThe range is based on the response times for six detectors. 20 ------- toward any of the test liquids, responding to all of them in less than 1 second. Detector A required over 9 hours to respond to diesel fuel but responded to all other test liquids in less than 20 minutes. Detector E responded to xylene(s), toluene, synthetic gasoline mixture, and commercial gasoline in 2 minutes or less; it required 6 to 7 hours to respond to n- hexane, diesel fuel, and jet fuel. Detectors C and F responded to all test liquids in under 10 minutes. i RESULTS OF ACCURACY AND RESPONSE TIME TESTING Accuracy, in general, is the degree of agreement between measured values and the true values. In these methods, the description of accuracy is quite different for qualitative-output versus quantitative-output detectors. Quali- tative accuracy in these methods is a measure of positive responses, where 100% accurate means the detector responded positively 100% of the time. Quan- titative accuracy is defined as "relative accuracy," following the convention used for continuous emission monitoring systems where 100% relative accuracy means the maximum expected bias (at the 95% confidence level) is +100% of the true value. Thus, qualitative and quantitative accuracy in these methods can- not be directly compared. Relative accuracy has no meaning for qualitative- output detectors; however, relative accuracy may serve as the prime comparator for quantitative-output detectors and may be determined in the laboratory against known test conditions or in the field versus a reference method. Bias is the term used for the systematic error inherent in a method, and it can be positive or negative. Precision is the degree of agreement between repeated measurements of the same parameter, and it reflects random errors that are unaffected by bias. Neither bias nor precision are applicable to qualitative devices, which only produce a binary ("on/off") signal. Detection time is the sum of lag time (the elapsed time from a detector's first contact with the test gas or liquid to the first detectable signal) and rise time (the elapsed time from a detector's first detectable signal in response to hydrocarbon liquid or gas to an output that is within 95% of full 21 ------- scale or "activated"). Fall time is the elapsed time from a hydrocarbon-free condition after exposure to hydrocarbon until a detector's output returns to within 95% of its original baseline level or there is no detectable signal output. Several parameters measured in this suite of tests describe accuracy; others describe time-related statistics that are important to the owner of underground storage tanks. Accuracy answers the question, "How close is the hydrocarbon value measured by a leak detector to the actual hydrocarbon level present?" Bias answers the question, "Is the detection device higher or lower than the real value?" Precision answers the question. "Is the test result repeatable and is the uncertainty in the test result large enough to adversely influence any decision process?" Detection time answers the question, "How fast will the detector signal an alarm after it contacts hydrocarbons?" Fall time answers the question. "How long will it take after a positive response for a sensor placed in a clean environment to cease its positive alarm sig- nal?" Answers to these questions are important because they indicate whether an alarm will occur at an acceptable performance level. Again, the following test results pertain to laboratory performance and not field performance. Vapor-Phase Detectors - Accuracy Testing Accuracy for qualitative-output vapor-phase detectors was defined in the test procedure as the number of positive responses expressed as a percentage of the total number of tests (five in this case). For quantitative-output detectors, the relative accuracy was calculated as the absolute mean diffe- rence between the measured value and the true value, plus the 2.5% error con- fidence coefficient for each series of five replicate tests, divided by the true value. It is expressed as a percentage of the reference (certified gas) standard concentration. Accuracy for qualitative-output detectors is neither conceptually nor numerically comparable to relative accuracy for quantitative- output detectors. 22 ------- Bias, for this method, was expressed as the signed (positive or negative) percent difference between the average measured value for a series of five replicate tests and the true value. Precision was expressed in terms of the percent coefficient of variation (CV), which is equal to the standard devia- tion for each set of five values, divided by the mean of the set, and multi- plied by 100. Each of six external vapor-phase petroleum hydrocarbon leak and release detection systems was tested with certified gas standards of benzene and isopentane. Four concentration ranges (approximately 50, 250, 500, and 1000 ppmv) were tested for both compounds. Benzene and isopentane were used as test gases because they are common components in petroleum products and many types of vapor-phase detectors are sensitive to them. Concentration levels were chosen based on preliminary information of the working concentration ranges for total hydrocarbons in which vapor release detection might be most appropriate. The test procedure (2) required an initial background test with ultrahigh-purity air to monitor the response of a test gas; record detection time, response value, and fall timej then repeat these steps until all of the tests were replicated five times for each test gas concentration, according to the matrix of 240 total tests in the project test plan (6). Each of the rep- licate tests was performed under equivalent test conditions. Duplicate analy- ses consisted of immediate repetition of test conditions for 10% of the tests chosen at random. Table 5 summarizes vapor-phase accuracy, bias, precision, and response time test results. In general, some of the vapor-phase detectors were very accurate, while other detectors were less accurate. Some of the vapor-phase detection systems were more accurate for a particular test gas, while others showed similar accuracy for both test gases. Some detectors were more accu- rate at higher test gas concentrations, while others were just as accurate at the lower test gas concentrations as they were at the higher concentrations. 23 ------- TABLE 5. VAPOR-PHASE ACCURACY, BIAS, PRECISION, AND RESPONSE TIME RESULT SUMMARY to Accuracy " Detector Test Gas Concentration (ppmv) Qualitative Accuracy (%r Relative Accuracy Bias (%)c (%) Precision (%) Detection Time (minutes) Fall Time (minutes) Qualitative Detectors B B B B B B B B C C C C C C C C quantitative D D D D Benzene Benzene Benzene Benzene Isopentane Isopentane Isopentane Isopentane Benzene Benzene Benzene Benzene Isopentane Isopentane Isopentane Isopentane Detectors Benzene Benzene Benzene Benzene 48 240 500 990 49 253 499 991 48 240 500 990 49 253 499 991 48 240 500 990 NTd NT NT 0 NT NT 0 0 NT 0 100 100 0 100 100 100 — NLe NL NL NL NL NL NL NL NL NL NL NL NL NL NL NL 44 -38 58 -55 61 -56 64 -55 NL NL .. NL NL NL NL NL NL NL NL NL NL NL NL NL NL 7 6 8 16 NDf ND ND ND ND ND ND ND ND ND 42.40 9.60 ND 24.00 42.00 42.40 8.00 8.00 8.00 16.60 ND ND ND ND ND ND ND ND ND ND 3.40 4.40 ND 3.20 7.80 3.20 8.00 8.00 8.00 8.00 ------- TABLE 5. (Continued) to Cn Detector3 Test Gas Qualitative Concentration Accuracy (ppmv) (%) Relative Accuracy Bias Precision Detection Time (minutes) Fall Time . (minutes) Quantitative Detectors (Cont.) G G G G G G G G Benzene Benzene Benzene Benzene Isopentane Isopentane Isopentane Isopentane 48 240 500 990 49 — — — 253 499 991 99 98 98 98 16 17 20 25 -99 -98 -98 -98 -13 -16 -18 -24 39 8 - 5 5 3 1 2 1 1.15 0.25 0.60 0.35 0.30 0.35 0.30 0.50 0.30 0.25 0.30 0.35 0.45 1.05 2.30 0.70 Tletector A was not tested because it did not respond during specificity tests. Qualitative Accuracy - For qualitative-output detectors, the number of positive responses to the test gas. expressed as a percentage of the number of tests (five) performed. A value of 100% means the detector responded appropriately for all five tests in the series; 0% means the detector failed to respond during any of the tests. Relative Accuracy - For quantitative-output detectors, the absolute mean difference between the measured value and the true value for the test gas. plus the 2.5% error confidence coefficient, expressed as a percentage of the true value. A value of 100% for relative accuracy means that, within the 95% confidence interval, the measurement bias can be expected not to exceed +100% of the true value. NT = not tested at this concentration. g NL = not applicable because the detector's output was qualitative. ND = not applicable because the test gas was not detected. ""Detection time and fall time values are only for test gas responses that were above baseline responses with ultrahigh-purity air. ------- TABLE 5. (Continued) to Detector Test Gas Concentration (ppmv) Quantitative Detectors (Cont.! D D D D ' E E E .E E E E E E E E E F F F F F F F F Isopentane Isopentane Isopentane Isopentane Benzene Benzene Benzene Benzene Benzene Benzene Benzene Benzene Isopentane Isopentane Isopentane Isopentane Benzene Benzene Benzene Benzene Isopentane Isopentane Isopentane Isopentane > 49 253 499 991 48 240 500 990 48 240 500 990 49 253 499 991 48 240 500 990 49 253 499 991 Qualitative Relative Accuracy Accuracy (%)B tt)c 46 54 29 42 687 370 150 173 687 370 150 173 1206 295 543 267 13 g 20 35 95 97 98 97 Bias (%) -33 -42 24 61 108 254 150 132 108 254 150 132 512 295 321 233 10 -6 -17 -34 -93 -96 -96 -97 Precision (%) 16 17 3 - 7 224 26 0 14 224 26 0 14 91 0 43 8 3 3 4 1 28 20 25 0 Detection Time (minutes) 6.40 8.00 4.00 6.40 0.25B 0.45 0.50 0.90 0.25s 0.45 0.50 0.90 0.55 0.30 0.65 2.40 0.25 0.30 0.25 0.25 0.25 0.30 0.30 0.25 Fall Tine (minutes) 7.20 8.00 7.20 8.00 0.258 0.80 1.00 2.15 0.25s 0.80 1.00 2.15 0.30 0.70 1.70 1.85 0.40 0.55 0.30 0.95 0.30 0.25 0.25 0.20 (Continued) ------- Vapor-phase Detectors A, B, and C were qualitative-output detectors. Detector A was not tested because it did not respond to any of the specificity test gases at 250 ppmv or to hexane at 500 ppmv. Its mode of operation was almost identical to Detector B. Detector B did not respond during accuracy testing to either test gas at any concentration tested. Detector C responded in all of the accuracy tests for 500 ppmv and 990 ppmv of benzene and for 253 ppmv, 499 ppmv. and 991 ppmv of isopentane. (The fact that Detector C re- sponded at 253 ppmv in these tests and did not respond at the same concen- tration during the specificity tests indicates that this concentration may be very close to the detector's detection limits.) Vapor-phase Detectors D, E, F, and G were quantitative-output detectors. Detector D's relative accuracy ranged from 29% to 64% for benzene and isopen- tane. - Relative accuracy (RA) values were similar for benzene and isopentane at all concentration levels. Detector E demonstrated poorer relative accuracy (RA = 150% to 1206%). Detector F was more accurate for benzene (RA = 9% to 35%) than for isopentane (RA = 95% to 98%). Detector G's relative accuracy values were more accurate for isopentane (RA = 16% to 25%) than for benzene (RA = 98% to 99%). Average bias values for quantitative-output vapor-phase detectors ranged from -99% to a high of 512%. In general, isopentane bias values tended to decrease as the test gas concentration increased. The best bias values were -6% and 10% for Detector F in benzene at 240 ppmv and 48 ppmv, respectively. Bias is not applicable to qualitative-output detectors., which produce only a binary ("on/off") output signal, and it was not calculated for vapor-phase Detectors A, B, and C. For Detector D, benzene bias was from -56% to -38%. For isopentane, this detector showed negative bias at 49 ppmv and 253 ppmv and positive bias at 499 ppmv and 991 ppmv. The bias magnitudes for both test gases were similar. For Detector E, bias values ranged from 108% to 512%. Isopentane bias values (RA = 233% to 512%) were higher than benzene bias values (RA = 108% to 254%). Benzene bias values for Detector F were from -34% 27 ------- to 10%, and the bias decreased with increasing test gas concentration! Detec- tor F isopentane bias results ranged closely from -97% to -93%. As with rela- tive accuracy results. Detector F bias results followed an almost opposite trend from bias results for Detector G. Detector G bias was -99% to -98% for benzene and -13% to -24% for isopentane. In general, all four quantitative-output detectors were very precise. All but one detector showed similar precision values for both test gases. The poorest precision values (91% and 224%) were observed for the lowest test gas concentrations with a single detector. All other precision values were under 50%. Precision is not applicable to qualitative-output detectors, which pro- duce only a binary ("on/off") output signal, and it was not calculated for vapor-phase Detectors B and C. Detector D's precision ranged from 3% to 17% CV. Precision values for Detector E ranged from 0% to 224% coefficient of variation (CV). showing less precision at the lowest test concentrations for both gases than at higher concentrations. As previously noted, test gases were at the low end of Detector E's operating range. Precision values for Detector F ranged from 0% to 28% CV. Even though this detector's best precision value was for 991 ppmv isopentane, benzene precision values were better than isopentane precision values overall. Detector G precision values were from 1% to 39% CV. Other than a 39% benzene precision value, values were under 10% for Detector G. This detector was slightly more precise for isopentane than for benzene. Liquid-Phase Detectors - Accuracy Testing Accuracy for qualitative-output liquid-phase detectors was defined in the test procedure as the number of positive responses for a particular test liquid at a particular thickness, expressed as a percentage of the total num- ber of tests in the series (five in this case). For quantitative-output liquid-phase detectors, the relative accuracy was calculated as the absolute mean difference between the measured value and the true value, plus the 2.5% 28 ------- error confidence coefficient for each series of five tests, divided by the true value. It is expressed as a percentage of the known thickness of hydro- carbon layer on water. As with vapor-phase detectors, accuracy, as defined in these methods, is neither conceptually nor numerically comparable for qualita- tive-output versus quantitative-output detectors. Bias is expressed as the signed (positive or negative) percent difference between the average measured value for each series of five replicate tests and the true value for the test liquid thickness. Precision is expressed in terms of the percent coefficient of variation, which is equal to the standard devi- ation for each set of five values divided by the mean of the set and multi- plied by 100. Each of six external liquid-phase petroleum hydrocarbon leak and release detection systems was tested with three thicknesses (0.04, 0.32, and 0.64 cm) each of simulated (laboratory-generated) gasoline mixture and commercial gaso- line. The simulated gasoline mixture will provide a uniform protocol for future testing; commercial gasolines frequently have differing product compo- sitions that may affect test results. The test procedure (2) required an initial background test in tap water to'monitor the response to each test liquid and thickness as a function of time, then repeating these steps until all of the tests were replicated five times for each liquid product and thickness in accordance with the matrix of 180 total tests given in the project test plan (6). Each of the replicate tests was performed under identical conditions. Table 6 summarizes the test results for accuracy, bias, precision, and response time testing for liquid-phase petroleum hydrocarbon detectors. Detector B was the only detector capable of providing quantitative out- put. Relative accuracy for this detector ranged from 1120% for the simulated 29 ------- TABLE 6. LIQUID-PHASE ACCURACY, BIAS, PRECISON, AND RESPONSE TIME "RESULTS SUMMARY _ Criteria Gasoline Type Accuracy Simulated Commercial Bias Simulated Commercial Precision Simulated Commercial Detection Simulated Time (minutes) Commercial Hydro- carbon Thickness (cm) 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 0.04 0.32 0.64 Detector A 100%a 100%a 100%a 100%a 100%a 100%a c c c c c c d d d d d d 6.18 0.83 0.58 1.37 1.30 0.70 B 11202?' 147%? 61? 444%? 87%? 42? 606% 114% 45% . 102% 57% 42% 59% 12% 9% 137% 18% 0% e <0.02 <0.02 e <0.02 <0.02 C 100%a ioo%a 100%a 100%a 100%a 100%a c c c c c c d d d d d d 2.73 0.32 0.32 2.42 0.23 0.27 D ioo%a 100%a 100%a 100%a 100%a 100%a c c c c c c d d d d d d 104.60 72.83 57.63 121.55 64.22 73.10 E 100%a ioo%a 100%a 100%a ioo%a 100%a c c c c c c d d d d d 13.30 0.08 0.32 8.13 0.13 0.12 F «: 100%a 100%a 0%* 100%a 100%a c c c c c . c d d d d d f 0.08 0.17. f 0.18 0.10 30 ------- TABLE 6. (Continued) Criteria Gasoline Type Fall Time Simulated (minutes) Commercial Hydro- carbon Thickness (cm) 0.04 0.32 0.6A 0.04 0.32 0.64 Detector A g g g g g g B h k k h k k C i i 2. ± i i D g g g g g g E g g g g g g F j k k j k k Qualitative Accuracy - For qualitative-output detectors, the number of positive responses to the test liquid, expressed as a percentage of the number of tests (five) performed. A value of 100% means the detector responded appropriately for all five tests in the series; 0% means the detector failed to respond during any of the tests. ^Relative Accuracy - For quantitative-output detectors, the absolute mean difference between the measured value and the true value for the test liquid, plus the 2.5% error confidence coefficient. expressed ae a percentage of the true value. A value of 100% for relative accuracy means that. within the 95% confidence interval, the measurement bias can be expected not to exceed +100% of the true value. Bias not applicable because device is qualitative. Precision was not applicable because the device was qualitative. elmmediate response when any hydrocarbon was detected. Not applicable because no response was recorded. ^Not applicable because the probe was destroyed during the test. immediate when activation occurs. ^Tfo deactivation after 24 hours in water. Will deactivate in air. applicable because there was no response. Immediate deactivation (fall time <1 second). 31 ------- gasoline mixture at 0.04 cm (1/64 in.,) to 42% at 0.64 cm (1/4 in.) for commer- cial gasoline. Poor accuracy results at the 0.04 cm (1/64 in.) thickness were expected because the manufacturer claimed a 0.32 cm (1/8 in.) lower detection limit. Accuracy values at 0.32 cm (1/8 in.) and 0.64 cm (1/4 in.) were much better, ranging from 147% to 42%. Detector B showed similar accuracy for both test liquids. Bias for external liquid-phase petroleum hydrocarbon detectors was only applicable to the one quantitative-output detector (Detector B) that was tested. Bias values for this detector ranged from 606% at 0.04 cm of simu- lated gasoline to 42% at 0.64 cm of commercial gasoline. As with the relative accuracy values for this detector, bias values for 0.32 cm and 0.64 cm test liquid thicknesses, which were equal to or higher than the detector's lower detection limit, were significantly better than the values at the 0.04 cm test liquid thickness. Precision for liquid-phase detectors was only applicable to Detector B because it was the only quantitative-output detector tested. Precision ranged from 137% at 0.04 cm (1/64 in.) to 0% at 0.62 cm (1/4 in.) of commercial gaso- line. As with relative accuracy and bias, precision values were much better for test liquid thicknesses that were above the detector's lower detection limit. Precision values at the 0.04 cm thickness were 59% and 137% for the synthetic gasoline mixture and commercial gasoline, respectively. Precision values at test thicknesses of 0.32 cm and 0.64 cm ranged from 18% to 0%. Only one of five qualitative-output detectors failed to respond to test liquids at each test thickness within 24 hours. Detector F, whose manufac- turer reported a detection limit of 0.32 cm (1/8 in.), failed to respond to either test liquid at 0.04 cm (1/64 in.). This same detector, however, did respond in all tests to both test liquids at the two higher test liquid thick- nesses. In other words, this detector was 0% accurate at a test liquid thick- ness of 0.04 cm but 100% accurate at test liquid thicknesses of 0.32 cm and 0.64 cm. 32 ------- Vapor-Phase Detectors - Response Time Testing Detection time and fall time results for vapor-phase detectors were given in Table 5. Vapor-phase detectors typically responded faster than their liquid-phase counterparts. The range of average detection times ranged from less than 15 seconds to 1 hour and 24 minutes. The vast majority of responses occurred in under 1 minute. Vapor-phase fall time values were smaller than detection time values. All fall time values were less than 1C minutes, and most values were under 1 minute. Liquid-Phase Detectors - Response Time Testing Detection time for liquid-phase detectors, as shown in Table 6, was gen- erally 2 hours or less. For all but one detector, responses occurred in less than 15 minutes if the test liquid was above the detector's lower detection limit. Detection times for liquid-phase Detectors B and F were very short (<1 to 11 seconds) at all test liquid thicknesses at which these detectors responded. These detectors did not respond at the 0.04 cm test liquid thickness, which was below the detection limit of both devices. Detectors A, C, and E responded at all three thicknesses of both test liquids in under 15 minutes. The longest average detection time for a set of five replicate tests from this group of detectors was 13.3 minutes for the simulated gasoline mixture at 0.04 cm. The shortest average detection time was 7 seconds for commercial gasoline at 0.64 cm. Detection times were much higher for test liquid thicknesses at 0.04 cm (1 minute, 22 seconds to 13 minutes, 18 seconds) than for 0.32 cm and 0.64 cm test liquid thicknesses (7 seconds to 1 minute, 18 seconds). There did not appear to be significant differences between simulated versus commercial gasoline or test liquid thick- nesses of 0.32 cm versus 0.64 cm. Detector D averaged from 57 minutes to 2 33 ------- hours for responses. Response times between tests with simulated gasoline were not significantly different from those for commercial gasoline. Fall time was generally not a significant factor for liquid-phase detec- tors. Fall time was not applicable to Detectors A. D. and E because they required replacement of a product-soluble component that was destroyed during contact with test liquid. Detector C would not return to an inactivated state when tested for fall time in water. This detector would return to an inacti- vated state if the probe was tested for fall time in air. Detectors B and F had fall times that were less than 1 second if the detectors responded to the test liquid. If these detectors did not respond to a test liquid, fall time was not applicable. 34 ------- REFERENCES 1. Eklund, B.M. and W.L. Crow. Survey of Vendors of External Petroleum Leak Monitoring Devices for Use with Underground Storage Tanks. EPA Contract No. 68-02-3994, Work Assignment 44. March 11, 1987. 2. Eklund, A.G., Achord,-R.D., Kienbusch, M.R., et al. Development of Procedures to Assess the Performance of External Leak Detection Devices: Performance Test Procedures, Draft Report. EPA Contract No. 68-03-3409, Work Assignment No. 02. March 11, 1988. 3. Eklund, B.M. and W.L. Crow. Development of Interim Performance Criteria for External Petroleum Leak Monitoring Devices Used with Underground Storage Tanks. Draft Technical Note. EPA Contract No. 68-02-3994, Work Assignment 70. November 14, 1986. 4. Eklund, A.G., Kienbusch, M.R., Achord, R.D.* et al. Development of Procedures to Assess the Performance of External Petroleum Leak Detection Devices: Draft Performance Test Procedures. EPA Contract No. 68-02- 3994, Work Assignment 92. March, 1987. 5. Kienbusch, M.R. and Randall, J.R. Device Acquisition Plan. Technical Note. EPA Contract No. 68-03-3409, Work Assignment 02, Task 02. August 1987. 6. Behrens, G.P. and Adams, K.M. Development of Performance Criteria for External Petroleum Leak Detection Devices: Quality Assurance Project "Plan and Test Plan for Laboratory Testing. EPA Contract No. 68-03-3409, Work Assignment No. 02. Plan for Laboratory Testing. August 1987. 35 ------- APPENDIX A MATHEMATICAL FORMULAS Vapor-Phase Detectors Relative accuracy—The relative accuracy (RA) of a set of data was calcu- ; lated as follows: RA = CC 100 (1) where: V = reference (theoretical) value; d = arithmetic mean of the difference of a data set. Equation 1.1; and cc = 2.5% error confidence coefficient (one tailed. Equation 1.2). Mean difference—The arithmetic mean of the difference (d) of a data set was calculated as follows: n d = - X d. (1.1) where: d. = measured response - theoretical response. Confidence coefficient—The one-tailed 2.5% confidence coefficient (cc) was calculated as follows: cc = t 0.975 (1.2) 36 ------- where: S = the standard deviation (n-1) of the data set; and t0.975 =2.5% (one-tailed) t value = 2.776 for n=5. Accuracy for qualitative-output detectors—For qualitative-output detec- tors, accuracy for a particular test atmosphere at a particular concentration was calculated as the number of positive responses expressed as a percentage of the total number of tests (five in this case) at that concentration with that test atmosphere, according to the following formula: A = 100 x (r /5)- (2) q p where: A = accuracy in percent; and r = number of positive responses. Bias—Bias is a measure of how much, on the average, the quantitative- output detector responds high or low with respect to the true (theoretical) response. Bias was calculated as follows: Bias = 100 x [(V - V )/V ] (3) o r r ! 5 where: V = the average observed value, r* Z V.; 0 D i=l 1 V. = the individual response to a test atmosphere; and Vr = the reference (theoretical) value. Detection time—Detection time was calculated according to the following formula: ET = T _ T (4) /, 1 37 ------- where: DT = elapsed detection time; T = clock time when the test atmosphere was first added to the test chamber; and T = clock time when the detector output went from an inactivated 2 state to an activated state for a qualitative-output detector or from a baseline reading to 95% of stable high level output for a quantitative-output detector. Fall time—Fall time was calculated according to the following formula: ** = T2 ~ Tl where: T?T = elapsed fall time; T = clock time when the test atmosphere was switched to ultrahigh- purity air; and T = clock time when the detector output went from an activated state to an inactivated state for a qualitative-output detector or from a high level reading to within 95% of stable baseline level output for a quantitative-output detector. Relative percent difference—The relative percent difference is a measure of variation between two observations. It is their absolute difference divided by their average, expressed as a percentage, and was calculated as follows: RPD = 200 x l(Vl - V2)/(V1 + V2)] (6) where: EPD = relative percent difference; V1 = larger value; and V = smaller value. 38 ------- Correlation coefficient—The correlation coefficient was calculated as follows: cc = V s ' s V xx yy where: cc = correlation coefficient; 2 S = n£x. - (Zx.) 2Q£ J- J- (7) x. = i x value; and x .th - y. = x y value. x Coefficient of variation (%) — The percent coefficient of variation is equal to the standard deviation for a set of values, divided by the mean of the set, times 100 and was calculated as follows: CV(%) = (S/X) x 100 (8) where: CV = coefficient of variation; S = standard deviation for n values (n-1 degrees of freedom) ; and X = the arithmetic mean of n values. Response factor (RF) — Response factors were calculated as follows: RF = C /C - (9) m t where: C = the vapor-phase detection system response (ppm-v, %, etc.); and m C = the concentration of the test atmosphere gas (ppm-v, %, etc.). 39 ------- Liquid-Phase Detectors . : Water-miscible substances content — The fraction of water-miscible sub- stances in commercial gasoline test product was calculated with the following equation for a 20-mL sample: (m - m.) where: w = water-miscible substances content; m = upper meniscus volume in mL; and m. = lower meniscus volume in mL. Area from immersing in water — The cross-sectional area of a probe was measured by determining the volume occupied by the probe when immersed in water. The following equation was used to calculate the cross-sectional area: a. = (V. - V )/1.27 (11) Q X p 2 where: a, = cross-sectional area in cm ; a V. = volume between marks without probe in mL; V = volume between marks with probe in inL; and P 1.27 = height of column of water displaced in cm. Product volume — Before any testing began, a product layer of accurately known thickness was formed on water in the test container. The volume of test product to add to the test container to create a desired thickness of product was calculated with the following equation: t x (a - a,) _ c d v = - ; - 1 - w 40 ------- where: v = volume of product to add to container in mL; t = desired product thickness in cm; 2 a = test container cross-sectional area (193.77 cm ); c 2 a, = estimated detector cross-sectional area in cm ; and w = test product water content (w = 0 for synthetic gasoline). Product thickness—The thickness of a test product layer on water was not measured .but was determined from the volume of product added to the test con- tainer, test container dimensions, and detector dimensions. The thickness of product on water was determined with the following formula: t = v/(a - a.) (13) c d where: t = test product thickness in cmj 3 v = volume.of test product in cm ; 2 a = container cross-sectional area (193.77 cm ); and c 2 a, = detector cross-sectional area in cm . d Output at 95% of stable high level output—Detection time for quantitative-output detectors was measured from the time the detector con- tacted test product until it reached 95% of its final stable output. The following formula was used to calculate the 95% of stable high level output, • based on the stable baseline level output and the stable high level output. HB = BL + (HL - BL) x 0.95 (14) where: HB = output at 95% of stable high level outputj BL = stable baseline output; and HL = stable high level output. Output within 95% of stable baseline output—Fall time for quantitative- output detectors was measured from the time the detector was no longer in 41 ------- contact with test product until it reached 95% of its final stable baseline output. The following formula was used to calculate the output within 95% stable baseline level output based on the stable baseline level output and the stable high level output. OB = BL + (HL - BL) x 0.05 (15) where: OB = output within 95% of stable baseline output; BL = stable baseline output; and HL = stable high level output. Relative percent difference—The relative percent difference is a measure of variation between two observations, neither of which is considered a refer- ence value. It is their absolute difference divided by their average. expressed as a percentage, and was calculated as follows: RPD = 200 x [(V, - V_)/(V.. + V.)] (16) where: KPD = relative percent difference; V- = larger value; and X H V0 = smaller value. £* e Coefficient of variation—The coefficient of variation indicates the relative degree of variation associated with two or more values. It is often reported as "Relative Standard Deviation." The coefficient of variation was calculated as follows: CV (%) = (S/X) x 100 (17) where: CV = coefficient of variation; S = standard deviation of n values (n-1 degrees of freedom); and X = mean of n values. 42 ------- Accuracy—Calculation for accuracy of liquid-phase detectors depended on the type of output that the detectors produced. Some detectors produced quantitative output from which the thickness of product on water was deter- mined. Other liquid-phase detectors only produced an alarm or other qualita- tive signal that signified that petroleum product was present. Quantitative-output liquid-phase detectors—Accuracy for quantitative- * output detectors is a function of systematic error (hias) and random error (precision). Accuracy was calculated as the sum of the absolute value of the arithmetic mean of differences, |d|, and the absolute value of the confidence coefficient, cc. Relative accuracy for liquid-phase detectors, which is the accuracy at a given hydrocarbon thickness divided by the hydrocarbon thickness and multiplied by 100, was calculated for each test product at every hydrocarbon thickness. Relative accuracy—The relative accuracy (RA) of a set of data was calcu- lated as follows: Jli_L_i££i x 100 r where: V = reference (theoretical) value; d = arithmetic mean of the difference of a data set. Equation 18.1; and cc = 2.5% error confidence coefficient. Equation 18.2. Mean difference—The arithmetic mean of the difference (d) of a data set was calculated as follows: d= - d (18.1) where: d. = measured response - theoretical. 43 ------- Confidence coefficient—The one-tailed 2.5% confidence coefficient (cc) was calculated as follows: cc = t ' — (18.2) cc ^.975 n where: S = the standard deviation (n-1) of the data set; and tn 0,c = 2.5% t value = 2.776 for n=5.» o • The relative accuracy was calculated for both commercial gasoline and synthetic gasoline at every test product thickness. Relative accuracy values were plotted against hydrocarbon thickness. Qualitative-output detectors — For qualitative-output detectors, accuracy for a particular test product at a particular thickness was calculated as the number of positive responses expressed as a percentage of the total number of tests at that thickness with that test product, according to the following formula: A = 100 x (r /5) . I P where: A = accuracy in percent; and q r = number of positive responses. P Bias — Bias is a measure of how much, on the average, the quantitative- output detector responded high or low with respect to the true (theoretical) response. Bias was calculated as follows: Bias = 100 x [(V - V )/V ] (20) o r r 44 ------- 1 5 where: V = the average observed value, •=• Z V.; o 5 i=1 i V. = the individual response to a test atmosphere; and Vr = the reference (theoretical) value. Bias at a particular thickness for a particular test product was the percent difference between the average detector output for a series of tests and the actual thickness of test product. In this calculation, the average detector output reading is used as the observed value (V ) in Equation 13, and the actual test product thickness is the reference value (V ). Bias was reported in percent. Detection time—Detection time was the elapsed time between introduction of the detector probe into the test container and when the detector reached a positive response. The nature of a positive response was dependent on whether the detector had quantitative or qualitative output; however, the calculation for detection time was the same for both types of detectors. Detection time was calculated according to the following formula: DT = T2 - TI (21) where: DT = elapsed detection time; TI = clock time when liquid was first added to test container; and T_ = clock time when detector output went from an inactivated state to £• an activated state for a qualitative-output detector or from a baseline reading to 95% of stable high level output for a quantitative-output detector. Fall time—Fall time was the amount of elapsed time between removal of test product from the test container and when the detector reached a negative response. The nature of a negative response was dependent on whether the 45 ------- detector had quantitative or qualitative output; however, the calculation for detection time was the same for both types of detectors. Fall time was calcu- lated according to the following formula: FT = T2 - TX where: FT = elapsed fall time; T = clock time when detector was removed from test container; and T = clock time- when detector output went from an activated state to an inactivated, state for a qualitative-output detector or from a high level reading to within 95% of stable baseline level output for a quantitative-output detector. Specificity — Determination of specificity was dependent on whether a detector's output was qualitative or quantitative. Quantitative-output detectors — Specificity for quantitative-output detec tors was defined as the ratio of detector output, or measured thickness, to the actual thickness of hydrocarbon test product expressed as a percentage. The following equation was used to calculate specificity for quantitative- output detectors: S = 100 x m,/t Q where: S = specificity in percent; m, = detector's output reading in cm; and d t = hydrocarbon thickness in cm (t = 1.27 cm). Specificity for qualitative-output devices was reported as "activated" if the detector responded within 24 hours. Otherwise, specificity was reported as "inactivated." 46 ------- GLOSSARY Accuracy - The degree of agreement between measured values and the true values. Accuracy in a single measurement reflects both systematic and random error. Accuracy in multiple measurements estimates of the same parameter, i.e., average accuracy, reflects systematic error, or bias. Activated - Refers to the state of a qualitative detector's response when indicating the presence of hydrocarbons. Bias - The systematic error inherent in a method. Bias may be positive or negative. Continuous Petroleum Hydrocarbon Detection Systems (Detectors) - A class of detectors that monitor in a constant, real-time mode, without interruption. Detection Time - Sum of lag time and rise time. Fall Time - The elapsed time after the test atmosphere is displaced by ultrahigh-purity air until its output returns to within 95% of its original baseline level or there is no detectable signal output. Intermittent Petroleum Hydrocarbon Detection Systems (Detectors) - A class of detectors that monitor on a. periodic basis. Interruptions may be due to sample transfer, sample analysis, system removal, installation, detection response time, etc. A continuous detector may be operated in an intermittent mode. Lag Time - The elapsed time from a detector's first contact with the test atmosphere or test liquid to the first detectable signal. 47 ------- Memory Effects - An effect typically encountered when tests using test atmospheres or test liquids of different concentrations or compounds are performed in succession. The effect is caused by residual hydrocarbon vapors from the previous test significantly interfering with the succeeding test. Non-activated - Refers to the state of a qualitative output detector's response when indicating that no hydrocarbons are detected. Petroleum Hydrocarbon Detection System (Detector) - Detection system (probes, lines, control bo*, readouts, etc.) for hydrocarbon compounds speci- fically designed or used for vapor-phase petroleum product detection. Precision - The degree of agreement of repeated measurements of the same parameter. Precision estimates reflect random error and are not affected by bias. Probe - Component of a detection system that must come into contact with petroleum vapor before the vapor can be detected. Qualitative Response - A type of detector response that indicates only the presence or absence of hydrocarbons without determining the hydrocarbon concentration. Quantitative Response - A type of petroleum hydrocarbon leak detection system response that quantifies the concentration of the hydrocarbon present. Relative Accuracy - The absolute mean difference between a group of mea- sured values and the true value, plus the 2.5% error confidence coefficient, divided by the true value. Relative accuracy is a measure of the maximum (upper 95% limit) expected bias for a series of measurements. Responses - The detector's indications of the presence or concentration of petroleum hydrocarbons. These can be qualitative or quantitative. 48 ------- Response Factor - The ratio of the detector response (ppmv, %, e'tc.) to * the test atmosphere or liquid concentration (ppmv, %, etc.). » Response Time - A general term that refers to the more specific terms of lag time and rise time, which together constitute detection time, and fall time. Rise Time - The elapsed time from a detector's first detectable signal in response to hydrocarbon vapor to an output that is 95% of full scale. Specificity (gas-phase) - The ability of a detector to detect a particular chemical or class of chemicals at 250 ppm-v. Specificity (liquid-phase) - The ability of a detector to respond to different substances. For the purposes of this procedure, these substances include commercial gasoline, synthetic gasoline, diesel fuel, jet fuel, n-hexane, xylene(s), and toluene. Test Atmospheres - The standard gases (Section 6.0) used to evaluate the performance of detectors. Test Chamber - The gas-tight or liquid-tight chamber where a detector is exposed to ultrahigh-purity air or water and hydrocarbon test gases or liquids. Test System - The arrangement of instrumentation and equipment required to perform this test procedure. For a larger glossary of terms consistent with the field of petroleum and hydrocarbon detection, consult the Survey of Vendors of External Petroleum s Leak Monitoring Devices for Use with Underground Storage Tanks Glossary (2). 49 ------- ------- ------- ------- |