Water Quality Exchange
XML Training Manual
for BEACH Monitoring Data
U.S. Environmental Protection Agency
Washington, DC
January 13, 2012
EPA Technical Support
1-800-424-9067
storet@epa.gov
Electronic copies of this document, along with many others,
are available at
http://water.epa.gov/grants_funding/beachgrants/datausers_index.cfm
NSCEP Publication No. 810B12001
-------
[This page intentionally left blank.]
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Table of Contents
Definitions v
Revision History vii
Introduction 1
Structure of BEACH WQX Data Submissions 1
Submission File 1
Header Document 2
Header 2
Payload 3
Payload Data Structure at a Glance 3
Data Elements 4
Organization 4
Project 5
Monitoring Location 6
Activity 7
Activity Description 7
Sample Description 9
Result 9
Result Description 10
Result Analytical Method 11
Result Lab Information 11
Trip Blank Activity 12
Activity Group 13
Updating and Inserting Data into WQX 13
Insert Process 14
Update Process 14
Deleting Data from WQX 15
Delete Process 15
Summary 16
Appendix A - Domain Values 19
Appendix B - Change Log 21
List of Figures
1. Structure of a Header Document 2
2. Overview of WQX XML schema structure for this tutorial 4
3. data elements 5
4. data elements 6
5. data elements 7
6. data elements 7
7. More detailed data elements 9
8. data elements 9
List of Figures (continued)
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
9. data elements 10
10. data elements 10
11. data elements for non-detect 11
12. data elements 11
13. data elements 12
14. Example trip blank activity 13
15. data elements 13
16. Delete XML file 16
List of Tables
1. Elements Used in Header of WQX Submission 2
2. General Rules that Apply to Updates of BEACH Monitoring Data 15
Version 3-08/06/2010 iv
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Definitions
Block—A group of data elements in an Extensible Markup Language (XML) file. A block
includes everything between a matching pair of open and close tags. For example, the
electronic address block could include the following two data elements: electronic
address text and electronic address type.
Central Data Exchange (CDX)—The U.S. Environmental Protection Agency's (EPA's)
electronic reporting mechanism that is used to submit legally acceptable data for various
programs into EPA databases.
Data Element—A data field, such as Project Name.
Extensible Markup Language (XML)—A format for transferring data by including a self-
describing tag on each data element that identifies the data element. An example of a self-
describing tag is . XML files are text documents that include tags for each
data element.
Header—The portion of the submission file that contains basic information about the data
submission.
Header Document—The .xml portion of a submission file that contains one header and one
payload. The header document is the only portion of a submission file that is allowed to
have an .xml extension.
Payload—The portion of the submission file that contains the Water Quality Exchange (WQX)
monitoring data being submitted.
Primary Component—A set of data elements or group of data elements in an XML file that
must be uniquely identified. A primary component includes everything between a
matching pair of open and close tags. In WQX, the four primary components are
, , , and
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Tag—A pair of identifiers used in XML files to label specific data elements. For example, a
project name of "Chesapeake Bay" would be identified by the tag
Chesapeake Bay. An open tag has no /, and a close tag
starts with a /.
Water Quality Exchange (WQX)—A data flow (or data transfer process) that uses an XML
schema to enable states, Tribes, the EPA, and others to share water quality monitoring
data.
Version 3-08/06/2010 vi
-------
Water Quality Exchange XML for BEACH Monitoring Data
Training Manual
Revision History
**Comments in blue reflect changes.
Date
2009-01-16
2009-01-27
2010-08-06
2012-01-13
Author
Paul Andrews
Paul Andrews
Paul Andrews
Paul Andrews
Description
Updated Delete XML file text and sample data
Added Revision History and Change Log, Minor Text Changes
Updated Insert-Update XML file, updated document links,
added text for clarification and to refer to query and solicit web
services, Updated Change Log on page 21 of this document
Modified link on cover page, deleted Overview of BEACH Data
Submission Process Changes section, Text Clarifications made
for Activity Groups, Project, Monitoring Location, Result
Description Sections, Changed ORGJD to WQXTEST,
Clarified Reusable Data Block definition, Updated Insert-
Update and Delete XML files to reflect WQXTEST ORGJD and
updated related screen shots, Updated Change Log on page
21.
Version 3-08/06/2010
vn
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
[This page intentionally left blank.]
Version 3 - 08/06/2010 vm
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Introduction
Welcome to the Beaches Environmental Assessment and Coastal Health (BEACH) Program's
Water Quality Exchange (WQX) Training Manual. WQX is a data flow (or data transfer process)
that uses an Extensible Markup Language (XML) schema to enable you to load data into the U.S.
Environmental Protection Agency's (EPA's) Storage and Retrieval (STORET) warehouse using
XML files. XML allows you to format the data being imported and ensures that the data are
consistent with the WQX XML schema. Once proper XML files are created that are consistent
with the schema, you can migrate BEACH monitoring data into the STORET warehouse.
This document will help you become familiar with the formatting protocol for submitting
BEACH monitoring data using the WQX XML schema and how to properly use WQX data
elements. This training manual covers the required data elements for an XML file based on the
WQX schema, but additional fields may apply in special cases. By the end of the tutorial, you
should be able to
• Identify WQX XML blocks and tags.
• Understand and be able to apply basic WQX business rules and data structure to BEACH
monitoring data.
• Create a BEACH monitoring data file to insert, update, or delete from the STORET
warehouse using the WQX data flow.
• Understand what activity groups and trip blanks are and how they can be documented
using XML.
This training manual includes both "WQX_BEACH_Demotest.xml" and
"WQX_BEACH_Demotest_Delete.xml" files, which contain sample data that may be used to
assist you in working through this tutorial. Please take a moment to find them on your computer
using Microsoft Windows Explorer or a similar navigation program. You can open these files in
a browser, such as Internet Explorer, or a text editor, such as Microsoft WordPad. Because you
will be accessing the "WQX_BEACH_Demotest.xml" and
"WQX_BEACH_Demotest_Delete.xml" in this tutorial, you should write down the full path
location on a piece of paper.
Structure of BEACH WQX Data Submissions
To properly submit, update, or delete BEACH monitoring data to EPA's STORET data
warehouse, the data must meet certain CDX and WQX structural and processing requirements.
All data must be submitted in XML format, have header information attached, and be contained
in a header document. The header document is zipped up in a ZIP file with any attached binary
objects (.pdf or .jpg) to create a submission file.
Submission File
The submission of BEACH monitoring data through CDX must consist of a submission file,
which is a ZIP file that contains a header document and any attached binary objects. The purpose
of creating a ZIP file is to not only consolidate the different portions of the submission into one
file, but also to increase the efficiency of the transmission due to the decrease in file size. CDX
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data
Training Manual
restricts the maximum allowable size of a submission file to 25 megabytes. The header document
is the only file within a submission that is allowed to have a .xml extension.
Header Document
A submission must consist of at least a header document, but may also contain other attached
binary files. The header document, which is the only file allowed to have a .xml extension, must
consist of one header and one payload. The header document will always begin with a tag, , followed by a tag that determines your
document ID and the path referencing the schema with which to validate your submission. The
header document must end with the closing tag of . Figure 1 shows the structure of
a header document.
-
+
+
Figure 1. Structure of a Header Document
Header
The header section of the header document provides basic information about the data
submission. The header can be inserted into an XML document containing the BEACH payload
by either copying and pasting it from another BEACH WQX document or by using a header
generation tool that is available from http://test.epacdxnode.net/helpdesk. Table 1 shows the
elements used in the WQX header section.
Table 1. Elements Used in Header of WQX Submission.
Element
Author
Organization
Title
Creation Time
Comment
Contact Information
Description
First and last name of the
individual generating the XML
document
Name of company, agency, or
individual generating the XML
document
Type of submission
Date/time when the document
was generated
Free text description of the
message contents
Name, mailing address, city,
state, zip code, telephone
number, and e-mail address
of person who may be
contacted with questions
about the submission
Example Value
Joe Smith
State X Department of
Environmental Quality
Must be "WQX"
2003-01-01T12:12:12 (Where date
is a valid XML date format string)
Joe Smith
123 Main Street
Portland, OR 97226
503-123-4567
Joe@deq.statex.gov
Required
Yes
Yes
Yes
Yes
No
Yes
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Payload
The payload portion of a data submission contains the WQX data that you are sending in XML
format. There are two different formats that the payload must conform to depending on the
purpose of your submission. If you are attempting to add data to the warehouse or modify data
that is already in the warehouse, you will use the Update-Insert schema. If you are attempting to
delete data from the warehouse, you will use the Delete schema. The payload section must
reference the appropriate schema as either WQX_WQX_v 1.0.xsd or
WQX_WQX_Delete_vl.0.xsd. Note that while the current version of the schema is version 2.1,
BEACH program monitoring data submissions may be submitted referencing either version 1.0
or 2.1.
The payload contains an attribute called "Operation," which determines whether your submission
is an "Update-Insert" or a "Delete." In XML, the following format appears: or . The end of the payload section
must contain a closing tag of . All of your monitoring data will be contained between
these tags. Specific monitoring data format for WQX will be discussed later in this tutorial.
Payload Data Structure at a Glance
To understand how WQX transmits your data in an XML file, it is important that you understand
how the data are structured. The WQX data structure follows the way organizations generally
report water quality monitoring.
• An organization is the group that collects water quality monitoring information.
• An organization may have one or more projects to collect data.
• Monitoring locations are places where monitoring activities occur.
• Monitoring activities are conducted by an organization's staff.
• Results are the data collected or produced from monitoring activities.
• Activity groups are collections of related activities, grouped as field sets, replicates,
subsamples, and quality control (QC) samples.
In the demonstration WQX file we will be working with (see Figure 2), as with all WQX XML
files, all of the data elements are included within the Organization group or root component. For
this tutorial, the organization is called RIDOH and the Organizationldentifier (ORG_ID) is
WQXTEST. Figure 2 illustrates how WQX is structured to follow monitoring steps. If you count
the tags of the different primary components, you will see that we will be working with the data
elements for one project, one monitoring location, two activities, and one activity group for this
organization.
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
- John Doe
Organization >RIDOH
WQX2007-01-03T12:43:18John Doe 401-555-1234
-
-
-
+
+•
+•
+•
+
+-
+
+•
+
+
+
Figure 2. Overview of WQX XML schema structure for this tutorial.
Data Elements
If you review an XML file with a browser, such as Internet Explorer, you can click on the "+"
sign next to the tag, which will provide more in-depth information. We will explore the
following components in this tutorial:
• Organization
• Project
• Monitoring location
• Activity
• Result
• Activity group.
Organization
The BEACH program, similar to other programs, requires that all organizations performing
monitoring activities be issued a unique identifier to differentiate it from other organizations
within the STORET warehouse. In the case of the BEACH program, this identifier is given to the
state agency performing the monitoring. The must be used every time
an organization wants to insert data into, update, or delete data from the STORET warehouse
using the WQX data flow.
All of the information in an XML file is contained within an organization. Within the
Organization root component, there are several primary data elements, including organization
address, telephone information, electronic address, organization address, project, monitoring
location, and activity.
Version 3-08/06/2010 4
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
The Organization root component has only a few simple business rules, as indicated in version
1.0 or version 2.1 of the Water Quality Exchange Flow Configuration Document1, which defines
WQX data services, approaches, and processes used to exchange information, and the Water
Quality Exchange Data Exchange Template2, which defines all WQX data elements and
associated business rules. For example, the Organization Identifier and the Formal Name of the
Organization fields are required. No other fields are required, although a few are conditionally
required if certain other information is provided. For example, if a telephone number is provided
for the organization, then the Telephone Number Type is required.
Project
Within the BEACH program, a project refers to an individual beach to be monitored. In other
words, the is really a beach identifier. Figure 3 shows the required
information for projects in the WQX XML schema. Within the primary component,
the data are stored between tags of related fields. For example,
• The unique project ID is RIBEACH-001, and the data are labeled with the
open and close tags.
• The project name is labeled by the tags.
• The project description is labeled with the tags.
If you use a text editor to view an XML file (as shown in Figure 3), you will see the
corresponding data information for each field between the open and close tags.
• RIBEACH-001Sandy BeachBEACH Act BeachID for linking to water quality sample
Figure 3. data elements.
The primary component has two required fields and one conditionally required field.
The two required fields are Project Identifier and Project Name. The Project Identifier, which can
be up to 35 characters long, can be reused to update or edit project data in WQX without
repeating all of the data fields in XML. Project Description is a conditionally required field,
meaning that it is required to describe the project, except under the condition that an attached
binary large object is provided in the primary component. A binary large object may
be a PDF or other file describing the project purpose in detail. The fields containing the file name
1 U.S. EPA (Environmental Protection Agency). 2006. Water Quality Exchange Flow Configuration Document,
Version 1.0. U.S. EPA, Office of Water, Washington, DC. November 15. Available at
http://www.exchangenetwork.net/exchanges/water/wqx.htm.
2 U.S. EPA (Environmental Protection Agency). 2006. WQX Data Exchange Template. U.S. EPA, Office of Water,
Washington, DC. September 13. Available at http://www.exchangenetwork.net/exchanges/water/wqx.htm.
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
for the binary large object and the text extension of the file are only required if the
header is inserted.
Monitoring Location
The primary component represents the stations used for monitoring at
your beach (project). As mentioned in the Project section of this tutorial, each project in the
BEACH program represents an individual beach. Each individual beach can be monitored in
multiple locations, which would be reported separately. However, it should be noted that there is
no way during input for a data submitter to associate monitoring locations with projects, other
than by specifying them in the Activity Description block for each activity, as shown in the
following section.
Next, expand and further expand and
to view the data between the tags for the fields of monitoring
locations. As shown in Figure 4, there is only one location for this beach, ID RIBEACH-001-1,
which is located at the western end of Sandy Beach. Its is
BEACH Program Site-Ocean. is the equivalent of what
Primary Station Type was in STORET, but unlike STORET, WQX no longer breaks out a
Secondary Station Type. The business rules for monitoring location require that the following
information is provided: an ID, a name, the type of location, and the geospatial information.
- RIBEACH-Q01-lSandy Beach lBEACH Program Site-OceanThis station is located at the western end of Sandy
Beach.
- 41.32-71.81100000NAD83
Figure 4. data elements.
Please note that some tags must contain a value from a limited list of options. For example,
BEACH Program Site-Ocean is one of a specified set of allowed values for
. The EPA has created Water Quality Exchange Domain Value
Lists, which are available using the query and solicit web services described in the Water Quality
Exchange Flow Configuration Document to show what the allowable values for tags requiring
specific choices. Appendix A contains a list of tags that have allowable values.
In the block, we can see that the location is geospatially
located at latitude 41.32 degrees, and longitude -71.81 degrees by the "Interpolation-Map"
method using the "NAD83" datum. As the business rules in the WQX Data Exchange Template
indicate, "Interpolation-Map" is the only Horizontal Collection Method that requires a Source
Map Scale Numeric value to be entered. Figure 4 shows that the scale of the map used to
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
determine the latitude and longitude was 1:100,000 by placing the integer "100000" between the
tags.
In addition to the , which is a required, reusable data block,
a reusable data block may be inserted in the block. The block is optional, but if used must contain a vertical
measure value, the vertical measure units, the vertical measure collection name, and the vertical
coordinate reference system datum name. The and
blocks are the first examples of reusable data blocks in this tutorial.
Activity
Each separate activity in the primary component represents a unique sampling or
measurement event. Expand one of the three blocks by clicking the "+" sign. As
shown in Figure 5, this reveals in-depth information about the ,
, , and two blocks. Of these four block types,
only is required.
+
+
+
+
Figure 5. data elements.
Activity Description
In the block, the project and monitoring locations are assigned to the
activity. As shown in Figure 6, we see that the unique activity, 01-23-456-78, has been
conducted for the project RIBEACH-001 and at the location RIBEACH-001-1.
- 01-23-456-78Sample-RoutineWater2006-02-02
-t- 2006-02-02
+ Mid water
+ 'DepthAltitudeReferencePointText>SURFACE
RIBEACH-001EPABEACHRIBEACH-001-l
-^/Activity Descriptions
Figure 6. data elements.
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
The data element is only conditionally required because there
are a few activity types that do not require a monitoring location, such as certain types of blanks
(which are quality control samples), which can be tied to specific activities using Activity
groups. The Water Quality Exchange Domain Values List names all activity types that can be
used in the data element and notes whether each one requires a monitoring
location to be specified.
The data element is required. As with monitoring locations, the Water
Quality Exchange Domain Values List can be used to determine if an analytical method must be
provided, based on the activity type code specified. For example, a routine sample requires the
entry of an analytical method, while a field measurement does not.
In addition to the , you must provide information in the following data
elements: , , , and
. Please note that the activity identifier must be unique across the
organization, which is a change from STORET. To accommodate this stricter requirement, WQX
allows the use of up to 35 characters to identify the activity. The Water Quality Exchange XML
Data Dictionary documents additional business rules concerning activities.
In addition to the normal WQX business rules, beaches users are asked to provide two project
IDs for each Activity. One project should be the Beach ID, while the other should be the
EPABEACH project. The tags, in this case, are given twice, as shown in
Figure 6. The EPABEACH project is used to allow for national-level queries to be generated to
pull only beach program data from STORET.
By expanding the , , and blocks, you can view additional time and depth information, as shown in Figure 7. For
example, you can see when an activity started and ended and at what depth an activity occurred.
Time-related information, such as (e.g., EST [Eastern Standard Time]) must
be an allowable value from the TIME_ZONE table in the Water Quality Exchange Domain
Value Lists (see Appendix A).
The same rule applies to the data element—the value for measure unit code
must be an allowable value from the MEASUREMENTJJNIT table in the Water Quality
Exchange Domain Value Lists. Please note that if the block is
used, the block may not be used, and vice versa. This is
because allows the user to specify a single measurement of
depth or height, whereas the indicates that a measurement
was taken over a range of depths. The Water Quality Exchange XML Data Dictionary provides
additional business rules concerning activity measures.
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
EST
20D6-D2-02ll:00:00EST
< Activity RelativeDepthName>Midwater
3ft
Figure 7. More detailed data elements.
Sample Description
The block is required for any activity that is some type of sample. Figure 8
shows an expanded view of the block. There are two requirements for the
sample description, if it is included. First, the block must be
provided. Second, the must contain a value from the WQX
Domain Values List. Please refer to the WQX Data Exchange Template for other business rules
for the block.
+
-
- ROUTINE
WQXTESTRoutine Sampling ^'MethodName>
500 mL sterile Nalgene bottle for beach water collection.Water Bottle
-
- BCHSAMPLEWQXTESTBeach Water SampleNalgene BottleOpaqueWet Ice (4 deg C)Samples stored in iced cooler while in
transit.
<^SampleDescription>
•t-
Figure 8. data elements.
Result
Although would seem to be a major block, it is a part of the primary
component, as shown in Figure 9. Because is not a primary component, the entire
must be submitted each time that a result is inserted or updated into the data
warehouse. By expanding one of the two blocks under , you can view the
details of that result.
Version 3-08/06/2010
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
+
+
-
+
+
+
+
Figure 9. data elements.
Result Description
The block, which is shown in Figure 10, is where results information is
stored. Most results fields require that allowable values be selected from the Water Quality
Exchange Domain Value Lists (see Appendix A), except for and
(not shown). For certain characteristics, the must also
be filled from the Water Quality Exchange Domain Value Lists. For example, the characteristic
"Flow, stream stage (choice list)" must have one of the following results: ABOVE NORMAL,
DRY, FLOOD, INTERSTITIAL, LOW, NO FLOW, or NORMAL.
It should also be noted that (Figure 10) is a conditionally required field
paired with (Figure 11). For most applications in the Beaches
user community, there are two cases: samples that produce a numerical value, and those that do
not. For samples that produce a numerical value in ,
must be blank. For samples that do not produce a numerical
value, must be blank, and must
contain a value, such as "Not Detected" or "Present Below Quantification Limit". Text values
may also be used in the field with similar business rules; however,
Beaches users usually report numerical result values. If
contains text, then the block (See Figure 13) must
contain the detection limit or quantitation limit of the analytical method or instrument used by
the laboratory. Figure 11 shows the for a case where
contains a value.
Fecal Coliform
- 13MPN/100mlFinal
Mean
Calculated
Figure 10. data elements.
Version 3-08/06/2010 10
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Not DetectedFecal ColiformFinal
Figure 11. data elements for non-detect.
The Water Quality Exchange Domain Value List for Characteristics also provides guidance on
which characteristics require the field be filled in. Chemical
characteristics usually require that a sample fraction be entered, but not always. Non-chemical
characteristics usually do not require that a sample fraction be entered.
If a result is reported, it is required that the be provided. This field may
be filled with "Final", "Preliminary", "Accepted", "Validated", or "Rejected". Results with a
status of "Preliminary" are not uploaded to the STORET Data Warehouse, and are not available
to the public. The is an optional data element that can be provided, if the
result is derived or calculated from other values. The example shown in Figure 10 shows that the
Result is a "Mean" value, while other examples of inputs for this field are "Daily Geometric
Mean", "30-day Geometric Mean", and "Single Sample Maximum". Numerical results also
require that the is entered. This field indicates whether the result value
is "Actual", "Calculated", "Control Adjusted", or "Estimated".
Result Analytical Method
The reusable data block is required only for certain activity types. In
general, all types of samples (except depletion replicate samples) require an analytical method to
be entered from the Water Quality Exchange Domain Value Lists. Field measurements never
require an analytical method. In the example file that was provided with this training guide,
allowable values from the Water Quality Exchange Domain Value Lists are used for the data
elements shown in Figure 12, which include , ,
and . However, if you input a non-domain value in ,
such as your state agency abbreviation, then you can define your own and
. Please refer to the WQXData Exchange Template for complete business rules
concerning the result analytical method.
9221-EAPHAFecal Coliform Procedure- Multiple-Tube Procedure
Figure 12. data elements.
Result Lab Information
The block, which is shown in Figure 13, contains information about the
laboratory that conducted the analysis.
Version 3-08/06/2010 11
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
XYZ Laboratories2006-02-03
- EST
- Method Detection Level (MDL)
- 2
MPN/100ml
Figure 13. data elements.
The data values and must be allowable values from the
Water Quality Exchange Domain Value Lists. The block also contains
a reusable data block called , or the detection limit of the
analytical method used by the laboratory. This detection limit or quantitation limit must be
provided if a of "Not Detected," "Present Above
Quantification Limit," or "Present Below Quantification Limit" is entered. The
must always be provided along with detection or
quantitation limit values. This field defines the type of detection or quantitation limit used in the
analysis. In this example, it is "Method Detection Level (MDL)" because the analyte was below
the level that could be detected by the method used.
Trip Blank Activity
Sometimes, you may want to provide information about QC samples, such as trip blanks. The
primary purpose of blanks is to trace sources of artificially introduced contamination, such as
those due to shipping or laboratory procedures. The blocks of a trip blank will contain
the results of this trip blank analysis.
As shown in Figure 14, a trip blank activity will look similar to other activities; however the
tag must say "Quality Control Sample-Trip Blank" and, as always, the
must contain a unique ID. Please note that the block
is included for a trip blank activity because it is a type of sample and must contain a
, but the is not required. Results of
the trip blank analysis are entered into each of the blocks. A final requirement for trip
blanks is that they must be associated with the other related samples by creating an activity
group. Examples of activity groups can be found at http://www.epa.gov/storet/wqx_products/
Activity_Group_Examples.pdf.
Version 3-08/06/2010 12
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
- 01-23-456-78-TBQuality Control Sample-Trip BlankWater
2006-02-02
-
-
+ Water Bottle
+
+
Figure 14. Example trip blank activity.
Activity Group
Activity groups are categories of activities and include field set, replicate, subsample, and QC
sample. For instance, to group a trip blank or other QC sample with other activities, you must
create an activity group. Within an block, activities are grouped by a unique
activity group identifier, and all of the included activities (e.g., 01-23-456-
78 and 01-23-456-78-TB) share a common , which must be an
allowable value from the Water Quality Exchange Domain Value Lists. These fields are shown
in Figure 15 for the case of a trip blank.
CBC-001-QCQC SampleQl-23-456-78
Ql-23-456-78-TB
Figure 15. data elements.
Updating and Inserting Data into WQX
The schema that WQX uses to perform inserts and updates of data is shared between both of
these operations because they are performed in a similar fashion. To perform either of these
operations, the header document must contain one header and one payload. The payload must
Version 3-08/06/2010 13
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
contain the tag for both of these operations and must
reference either WQX_WQX_vl.0.xsd or WQX_WQX_v2.1.xsd as the schema. The WQX
system will automatically determine whether a BEACH monitoring data submission is being
updated or inserted.
Insert Process
In the case of inserting BEACH monitoring data into WQX, the data being inserted are new, and
therefore not already present in WQX. All insert files must contain the complete root component
for the entire hierarchy: Organization. Only one organization may be contained in the
submission. Additionally, a BEACH monitoring data submission to WQX must contain at least
one of the four following primary components:
•
•
•
•
These primary components must be uniquely identified within an organization; however, results
do not need to be uniquely identified within an organization. These are only unique within a
parent activity. Within an activity, you may insert as many BEACH monitoring results as
necessary.
Update Process
In the case of updating BEACH monitoring data in WQX, the data being updated are already
present in WQX. As with the insert process, an update submission file must contain only one
complete organization, which is the root component, and at least one of the four following
primary components:
•
•
•
•
To be updated, these primary components must already be uniquely identified within an
organization. The primary component level is where all BEACH monitoring data must be
updated. This is because children of the primary components are not uniquely identified within
the organization. For instance, if you want to update a Fecal Coliform result of Activity 2, you
must update all of Activity 2, because Activity 1 may also contain a Fecal Coliform result. In
other words, you must be specific about which Activity's results you are intending to update, and
this is done by updating the entire Activity.
Certain general rules apply to updates of BEACH monitoring data that are already in WQX.
Following these rules will help you avoid unintended changes to your data during updates.
Table 2 provides a list of these rules.
Data that contain errors will cause the load to fail at the primary component level. This means
that if a result fails because of an error, the entire parent activity will fail to load, along with all
associated child results. Other activities with no errors will load fine because all successful loads
Version 3-08/06/2010 14
-------
Water Quality Exchange XML for BEACH Monitoring Data
Training Manual
are immediately committed into the database. The following are two approaches that you may
use to correct failed loads to WQX:
• Correct the data that failed and resubmit the entire job. Those records that successfully
loaded will be automatically treated as an "Update" and the failed record will be treated
automatically as an "Insert."
• Correct the data that failed and submit only the failed portions. The advantage of this
approach is that your file may be significantly smaller; therefore, it will process more
efficiently.
Table 2. General Rules that Apply to Updates of BEACH Monitoring Data.
Intended Update Action
Update only the primary component, but not child
data. Example: Update activity, but leave
associated results alone.
Update only the primary component and delete child
data. Example: Update activity and delete all
associated results.
Update all results in an activity.
Update only one of multiple results in an activity.
Update one of multiple results in an activity and
delete all other results.
Rule
Provide complete primary component tags
(e.g., activity tags), but do not include results
tag.
Provide complete primary component tags
(e.g., activity tags) and include an empty
Results tag.
Provide complete activity tags and complete
tags with updated information for all results in
that activity.
Provide complete activity tags and complete
tags with information for all results in that
activity with updated information for the one
result to be updated.
Provide complete activity tags and updated
information for only the result to be updated. Do
not provide information for results to be deleted.
Deleting Data from WQX
The schema that WQX uses to perform deletes of data allows the deletion of data for an entire
primary component. In order to perform a "Delete," the header document must contain one
header and one payload. The payload must contain the tag , and
must reference either WQX_WQX_Delete_vl.0.xsd or WQX_WQX_Delete_v2.1.xsd as the
schema.
Delete Process
If you discover that BEACH monitoring data are no longer valid, it is necessary to remove that
data from WQX by performing a delete. The WQX Delete schema allows you to extract data
from WQX at the primary component level. For example, if you want to delete an entire project,
monitoring location, activity, or activity group, you would use the Delete schema. However, if
you want to remove one result from within an activity, you should follow the "Update-Insert"
process instead.
The payload section of the header document for a deletion from WQX must contain the
and one or more of the following within the
tags: , ,
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
. The is included to provide context for the
delete. Only one organization can be provided. Organizations cannot be deleted from WQX
using the Delete schema. The primary components can be repeated, thus allowing the deletion of
several projects, monitoring locations, activities, or activity identifiers at one time. Figure 16
shows an example of a complete WQX XML file used to delete BEACH monitoring data for the
project RIBEACH-001 and the monitoring location RIBEACH-001-1 from the organization
WQXTEST.
- John Doe
Organization >RIDOH
WQX2007-01-03T12:43:18John Doe 401-555-1234<^ContactInfo>
-
-
- WQXTESTRIBEACH-001RIBEACH-001-l
Figure 16. Delete XML file.
Please note that whenever a parent primary component is deleted, all child data are also deleted
The delete XML file, shown in Figure 16, will also remove the activities that are associated with
the project and monitoring location being deleted. Other examples of this include the following:
• Attached Binary Objects are deleted when a parent project, monitoring location, or
activity are deleted.
• Results are deleted when the parent activity is deleted.
• Activities are deleted when their parent project or monitoring location is deleted. The
exceptions to this are when an activity is related to more than one project or monitoring
location, and also, as in the case of QC samples, when it does not relate to any monitoring
location.
Summary
The purpose of this tutorial is to help you understand the data submission format and XML tags
and blocks that are required to create a WQX XML file to submit BEACH monitoring data to the
STORET warehouse. In addition, the tutorial illustrates how to handle trip blanks and activity
groups. The information contained in the tutorial is based on the Water Quality Exchange XML
Flow Configuration Document, the WQX Data Exchange Template, and the Water Quality
Exchange Domain Values Lists, which are available through the query and solicit web services
described in the Flow Configuration Document, and on the National Environmental Information
Exchange Network's Web site at http://www.exchangenetwork.net.
Version 3-08/06/2010 16
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
Based on this introduction to WQX and XML, you can determine the best way to generate an
XML file for your BEACH monitoring data. Most users will use Oracle, Visual Basic, or other
software programs to generate XML files directly from their data system. For advice on this
process or for more information, please call the STORET hotline at 1-800-424-9067, or you can
e-mail support staff at storet@epa.gov.
Version 3-08/06/2010 17
-------
Water Quality Exchange XML for BEACH Monitoring Data Training Manual
[This page intentionally left blank.]
Version 3-08/06/2010 18
-------
Water Quality Exchange (WQX) XML
Training Manual
Appendix A- Domain Values
The following table shows XML tags and the allowable values tables that are associated with
them. For example, for the , you are required to select air, water, biological
tissue, soil, sediment, or other. The table below provides the domain table and domain field
names that must be used to fill each data element XML tag. The
"WQX.GetDomainValueByElementName" Web service method allows you to query or solicit
current domain value lists. These services will return an XML file in the format of the schema
WQX_DomainValues_vl.0.xsd or WQX_DomainValues_v2.1.xsd referred to in the Water
Quality Exchange Flow Configuration Document. An example XML file that would be returned
by the Web service can be found at
http://www.exchangenetwork.net/schemaAVQX/2AVQX_GetDomainValues_Example_v2.1.xml.
XML returned by the query and solicit web services may be transformed into table format using
DomainValuesCreator, which is available from ftp://ftp.epa.gov/storet/wqx/ref_tables/. However,
it should be noted that the DomainValuesCreator tool does not work on XML files returned from
the version 1.0 queries.
Data Element XML Tag
ActivityGroupTypeCode
ActivityMediaName
ActivityMediaSubDivisionName
ActivityRelativeDepthName
ActivityTypeCode
AddressTypeName
CharacteristicName
Country Code
County Code
DetectionQuantitationLimitTypeName
ElectronicAddressTypeName
HorizontalCollectionMethodName
HorizontalCoordinateReferenceSystem
Datum Name
MeasureQualifierCode
MeasureUnitCode
Methodldentifier
Monitoring LocationTypeName
ResultDetectionConditionText
ResultLaboratoryCommentCode
ResultMeasureValue (must match a
domain list value if the
CharacteristicName ends with the phrase
"Choice List")
ResultSampleFractionText
ResultStatusldentifier
ResultTemperatureBasisText
ResultTimeBasisText
WQX.GetDomainValueByElementName
Service Parameter
ActivityGroupType
ActivityMedia
ActivityMediaSubDivision
ActivityRelativeDepth
ActivityType
AddressType
Characteristic
Country
County
DetectionQuantitationLimitType
E lectron icAddressType
HorizontalCollectionMethod
Horizontal CoordinateReferenceSystem
Datum
MeasureUnit
Monitoring LocationType
ResultDetectionCondition
ResultLaboratoryComment
ResultMeasureValuePickList
ResultSampleFraction
ResultStatus
ResultTemperatureBasis
ResultTimeBasis
Domain Table
ACTIVITY_GROUP_TYPE
ACTIVITY_MEDIA
ACTIVITY_MEDIA_SUBDIVISION
RELATIVE_DEPTH
ACTIVITY_TYPE
ADDRESS_TYPE
CHARACTERISTIC
COUNTRY
COUNTY
DETECTION_QUANT_LIMIT_TYPE
ELECTRONIC_ADDRESS_TYPE
HORIZONTAL_COLLECTION_METHOD
HORIZONTAL_REFERENCE_DATUM
RESULT_MEASURE_QUALIFIER
MEASUREMENT_UNIT
ANALYTICAL_METHOD
MONITORING_LOCATION_TYPE
RESULT_DETECTION_CONDITION
RESULT_LAB_COMMENT
CHARACTERISTIC_PICK_LIST_VALUE
SAMPLE_FRACTION
RESULT_STATUS
RESULT_TEMPERATURE_BASIS
RESULT_TIME_BASIS
Domain Field
AGTYP_NAME
ACMED_NAME
AMSUB_NAME
RELDPTH_NAME
ACTYP_CD
ADDTYP_NAME
CHR_NAME
CNTRY_CD
CNTY_FIPS_CD
DQLTYP_NAME
EATYP_NAME
HCMTH_NAME
HRDAT_NAME
RMQLF_CD
MSUNT_CD
ANLMTHJD
MLTYP_NAME
RDCND_NAME
RLCOM_CD
CPLVAL_CD
SMFRC_NAME
RESSTA_NAME
RTMPB_NAME
RTIMB_NAME
Version 3-08/06/2010
19
-------
Water Quality Exchange (WQX) XML
Training Manual
Data Element XML Tag
ResultValueTypeName
ResultWeightBasisText
SampleCollectionEquipmentName
SampleContainerColorName
SampleContainerTypeName
SampleTissueAnatomyName
SampleTissueTaxonomicName
StateCode
StatisticalBaseCode
TelephoneNumberTypeName
ThermalPreservativeUsedName
TimeZoneCode
Tribal Code
Vertical CollectionMethodName
Vertical CoordinateReferenceSystem
Datum Name
WQX.GetDomainValueByElementName
Service Parameter
ResultValueType
ResultWeightBasis
SampleCollectionEquipment
SampleContainerColor
SampleContainerType
SampleTissueAnatomy
Taxon
State
StatisticalBase
TelephoneNumberType
ThermalPreservativeUsed
TimeZone
Tribe
Vertical CollectionMethod
VerticalCoordinateReferenceSystem
Datum
Domain Table
RESULT_VALUE_TYPE
RESULT_WEIGHT_BASIS
SAMPLE_COLLECTION_EQUIP
CONTAINER_COLOR
CONTAINS R_TYPE
SAMPLE_TISSUE_ANATOMY
TAXON
STATE
RESULT_STATISTICAL_BASE
PHONE_TYPE
THERMAL_PRESERVATIVE
TIME_ZONE
TRIBE
VERTICAL_COLLECTION_METHOD
VERTICAL_REFERENCE_DATUM
Domain Field
RVTYP_NAME
RWBAS_NAME
SCEQP_NAME
CONCOL_NAME
CONTYP_NAME
STANT_NAME
TAX_NAME
ST_CD
RSBAS_CD
PHTYP_NAME
THPRSV_NAME
TMZONE_CD
TRB_CD
VCMTH_NAME
VRDAT_NAME
Version 3-08/06/2010
20
-------
Water Quality Exchange (WQX) XML Training Manual
Appendix B - Change Log
Released 1/16/2009
o Corrections made in example Delete XML file to add tags.
o Delete Process text preceeding Figure 16 was modified to mention the
tags.
o Figure 16 was replaced with a screen capture from the corrected Delete XML file.
Released 1/27/2009
o Added a Revision History Table.
o Added Appendix B - Change Log.
o Changed Appendix A header to add "Domain Values" to the title and modified
the first sentence in Appendix A to state that the table contains domain values
tables, not the values themselves.
Released 8/6/2010
o Page 2 - Corrected text describing maximum allowable size of a submission file
to indicate the CDX imposed restriction of 25 megabytes.
o Pages 4, 6, 14, and 16 - Clarifications were made to note that either schema
version 1.0 or 2.1 may be used for submissions.
o Page 5 and 6 - Figure 2 and Figure 3 were updated to reflect correct number of
projects represented in Insert-Update XML file.
o Pages 6, 7, 10, 12, and 17 - Corrected the term Water Quality Exchange Data
Dictionary to WQX Data Exchange Template.
o Pages 7 and 17 - Changed text to reference the query and solicit web services that
should be used to retrieve the most up to date domain values lists. This text
replaced references to a static domain values table available on a website because
the static table is not routinely kept up to date.
o Page 8 - Figure 6 was updated to correctly represent the use of the EPABEACH
project for BEACH Program monitoring data submissions.
o Page 9 - Text was added to describe the correct use of the EPABEACH project in
each BEACH Program monitoring activity submitted.
o Page 11 - Figure 11 was updated to more correctly show the
data elements used when a non-detect is reported.
o Page 11 and 12 - Clarifying text was added to describe the use of the
and data elements.
o Page 15 - Clarification of text was made to better describe how the update process
works.
Version 3-08/06/2010 21
-------
Water Quality Exchange (WQX) XML Training Manual
o Page 19 - Updated text in Appendix A to reference both the schema version 1.0
and 2.1.
o Page 19 - Updated text in Appendix A to reference the domain values example
available for version 2.1 of the schema and provided a link to download the
DomainValuesCreator tool used to transform XML format query results to tabular
format.
• Released 12/13/2012
o Cover Page - Changed location of electronic copy of this document from
http://water.epa.gov/storet to
http://water.epa.gov/grants_funding/beachgrants/datausers_index.cfm.
o Page v - Clarification made to definition of Reusable Data Block.
o Page 1 - Deleted Overview of BEACH Data Submission Process Changes
including Table 1 because it is no longer a recent change to the submission
process.
o Page 2 - Changed Table 2 to Table 1.
o Page 3 - Revised definition of Activity Groups for clarity.
o Page 3 - Change ORG_ID to WQXTEST to allow submission of XML file for
training purposes. Clarified that the Organization and ORG_ID are RIDOH and
WQXTEST, respectively.
o Page 5 - Changed text to indicate that the Project Description is a conditionally
required field, rather than optional, since it is required unless an attached binary
large object describing the project is provided. Text describing the use of the
binary large object was slightly modified for clarity.
o Page 6 - Inserted language noting that monitoring locations are not associated
with projects by the data submitter, other than by being associated with each
individual activity.
o Page 9 - Replaced Figure 8 showing change of ORG_ID from 21RIBCH_WQX to
WQXTEST.
o Page 11 - Inserted language describing the use of the data
element.
o Page 15 - Changed Table 3 to Table 2.
o Page 16 - Change text to reflect change of ORG_ID from 21RIBCH_WQX to
WQXTEST.
o Page 16 - Replaced Figure 16 showing change of ORG_ID from 21RIBCH_WQX
to WQXTEST.
Version 3-08/06/2010 22