WQX Data Submittal Instructions
April 2011
United States Environmental Protection Agency
Office of Water
1200 Pennsylvania Avenue, NW
Washington, DC 20460
-------
SUBMITTING WATER QUALITY DATA to WQX/STORET:
SUBMISSION METHOD SELECTION INSTRUCTIONS
Monitoring ambient surface water and ground waters including beaches, rivers, lakes, bays and
streams is the responsibility of states territories and tribes. Organizations, tribes and government
entities have a variety of options available to them in submitting and accessing chemical, physical,
biological and habitat data and associated metadata. This document overviews different options for
submitting water quality data, to assist users in making their decision as to which option is best for
them (Figure 1).
Water quality data can be submitted to the water quality exchange (WQX) in multiple ways:
• via Exchange Network Nodes or Node Clients (Option 2 & 3)
• via the provided WQX Web application (Options 1).
Regardless of which option you select, you will also need:
1. An EPA WQX/STORET organization ID, (e.g., 21NYBCH), Organization IDs are assigned
by the EPA STORET team upon the request of the user. To obtain an Organization ID, send
an email to the STORET helpdesk STORET@epa.gov, or contact the WQX help desk 1-800-
424-9067.
2. An EPA NAAS account ID (usually your email address) and password. To request an NAAS
account, send an email to the STORET helpdesk (see 1, above).
For a more detailed explanation of the WQX network, review the WQX Flow Configuration
Documents (FCD) at: http://www.exchangenetwork.net/exchanges/water/wqx.htm
[For user support, contact 1- 800-424-9067 or STORET@epa.gov ]
Figure 1.
Organization
National
STORET
Warehouse
WQX/STORET
Submissions
Node Client/
State Node
XML
Organization
-------
Option 1 - Using WQX Web
Unlike the other options, WQX Web converts text file submissions into XML script allowing
organizations unable to generate XML script on their own to submit water quality monitoring data to
the STORET Warehouse. For more information on using WQX Web, please access the following
URL: http://www.epa.gov/storet/wqx/wqxweb downloads.html.
Option 2 - Using Node Client Software
Compared to other available options, submission through the Node Client application is more
suited for larger flow volumes. To use the Node Client software you will need a NAAS account
associated with your Organization ID (See page 1). If you already have an Org ID and NAAS ID,
download and install NodeClientLite2™ software (add icon to Desktop); e.g., from
http://www.windsorsolutions.biz/nodeclient2007/ (NodeClientLite2™) [For user support, contact
Node Support (nodehelpdesk@epacdx.net); 888-890-1995]. For additional guidance in submitting
data using node client software, please see Appendix A of this document. If you decide not to
use a State Node or Node Client, use Option 1.
Option 3 - Using a State Node
You can either obtain your own Node software or use a state node. Use of a Node assumes
you have a local data management system from which you can create XML
1. Contact your state Node Administrator to determine what it would take to use your state node to
submit data. For your state Node administrator's contact information see:
httj2i//ww^ (in the map, click on the state).
2. If you decide to use the state Node, your Administrator will create a National
Authentication & Authorization System (NAAS) account. You will need to contact the
EPA to have that account associated with your CDX account, contact
nodehelpdesk@epacdx.net 888-890-1995 option 2; and your WQX/STORET account (see page
1).
-------
Appendix A. Submitting Water Quality Data UsingNodeClient2 ™ Software
If you decide not to use the State node (option 3), you have the option of using the Node Client
software. Regardless of which option you select, you will also need:
1. An EPA NAAS account ID (usually your email address) and password, to request an
NAAS account, send an email to the STORET helpdesk (see part 3).
2. An EPA WQX organization ID, (e.g., 21NYBCH), Organization IDs are assigned
by the EPA STORET Team upon the request of the user contact the WQX help
desk (see part 3).
The STORET Helpdesk will pre-register you to the CDX Help Desk as a user for the notification
flow to WQX. You will receive an e-mail from EPA CDX HELPDESK with instructions on
completing your registration.
A. If you already have a CDX ID and password, the provided credentials (username
and password) can be used to submit to both Node test and Node production
B. If you are new to CDX: see the following instructions
Once registered, you will be directed to https://cdx.epa. gov/SSL/CDX/login.asp where your available
account profiles will be listed. Click on the link for the type of file you want to submit. You must
have an Org ID to be able to submit to WQX. An Organization Identifier (Org ID) is a unique
identifier to distinguish you within the WQX system and the STORET Warehouse (e.g.
ALO_WQX for the Alliance For a Living Ocean or 21ARIZ_WQX for Arizona Department of
Environmental Quality). To request an NAAS account, send an email to the STORET helpdesk
storet@epa.gov, and provide the following information.
• the NAAS accounts that you want to have access to your Organization ID
• First name
• Middle name
• Last Name
• Prefix (Mr/Mrs./Ms)
• WQX Organization ID
• WQX Organization Name
• Mailing Address 1
• Mailing Address 2
• City
• State
• Zip/Postal Code
• E-mail Address
• Phone Number
• NAAS accounts are assigned by Node Administrators or by the CDX Help Desk
For users of node clients (who don't have a Node Administrator), NAAS accounts are
assigned by the CDX Help Desk on an individual basis
Disclaimer: NodeClientLite2™ was developed by Windsor Solutions. This document does
not convey and should not be interpreted as conveying official EPA approval, endorsement
or recommendation of this application.
-------
Accessing the NodeClientLite2™ Application
The NodeClientLite2™ application has a two way flow, allowing users to submit data (in XML
format) and retrieve information from WQX i.e. documented domain values. NodeClientLite2™ also
provides users with a record of their submitted data. This reference of submitted data, in addition to
access to domain values, may be helpful in determining the cause of error messages. To obtain the
ability to interact with WQX web services as defined in the flow configuration document (FCD
http://www.exchangenetwork.net/schema/WQXy2/WQX FCDv2.1 .pdf) follow the steps below.
Note: although the file below is v2.0, the vl.O domain value and transaction ID services are still
supported, and this file works with both.
Obtaining All Services Available From STORET
Click on the NodeClientLite2™ Application from its location on your desktop, and select the open
icon. On the STORET FTP Site download and save (ftp://ftp.epa.gov/storet/wqx/wqxservices.xml).
I. Save the URL as file type "All files".
II. Name the downloaded file wqxservices.windsor.
III. Save the file to a location on your hard drive.
IV. In the NodeClientLite2™ application (Figure 2), under "My Preferences" click on
Services .
Right click on the ' Update' button.
-------
VI. Choose 'Import previous version file...'
VII. Navigate to where you saved the wqxservices. Windsor file.
VIII. Set files of type to Service Lite File.
IX. Click "Open".
You should now have all of the WQX services.
Do not click 'Update' again, or you will have more items, unrelated to WQX, in your list of services.
Inserting Your Username and Password Into NodeClientLite2™
Step 1. Use your NAAS ID and password to log in.
To log into the NodeClientLite2™ application, under My Preferences select Application (Figure 3).
Figure 3.
-------
Enter your credentials and select save (figure 4).
Figure 4.
Step 2. Connect to WQX through CDX.
In the upper left hand corner, under Things I can do of the application, note that Connect To Node is
in bold text, signifying an active option, while Get Data, Upload Documents and Download
Documents are in grey text signifying inactive options (Figure 5). The node is not connected to an
endpoint if these options are inactive, meaning data cannot be submitted or received.
Figure 5.
-------
You can obtain the appropriate endpoint to submit and receive data by accessing the exchange
network page. For more information about endpoints see the provided link
http://www.exchangenetworkwiki.com/wiki/index.php/Node Endpoints at U.S. EPA's Central Da
ta Exchange.
After accessing U.S. EPA's Central Data Exchange page, under Node Endpoints you will find a
section titled Water Quality Exchange (Figure 6). Under the Water Quality Exchange section are
links labeled Node 2.0 Test, Node 2.0 Production, Node 1.1 Test and Node 1.1 Production. Both
the 1.1 and 2.0 versions are options for you to use, however this tutorial will use the 1.1 version.
You must make one successful test submission from your endpoint, through Node Test before
being permitted to submit through Node Production.
Figure 6.
Toxic Release Inventory (TRI) Exchange
Mode 2 Q Tesi • Not yet available fen use with Mode 2 0
Mode 2 0 Production - Not yet available for use with Node 2.0
Node 1 1 Test - imps i«i epacdxrode nK'cdx(servtcesf'leiwcirtwsdl3
Node 1 1 Taat • Mips "tssl epacdxnode n«.'cdit.'3ejvices/Nelwort'.NodePonType_Vl03
Node 1 1 Production- imps -vcdxriod* «pa gowcdi/seiMces/IJetworkModePorlType^ViOS
Water Quality Exchange ( WOX)
Node 2 0 Test - mips '"trsi ef»c4xnrJeiwoikNo
-------
Step 3. Obtaining a Test connection endpoint.
To connect your node to the EPA node, locate the New Connection Pane in the window. Ensure that
the correct version is selected (in this case Original ((vl.l) is used) and enter the test endpoint for
Node 1.1 (https://test.epacdxnode.net/cdx/services/NetworkNodePortType VI0) provided by the
exchange network page
(http://www.exchangenetworkwiki.com/wiki/index.php/Node Endpoints at U.S. EPA's Central D
ata Exchange).
Click Connect (Figure 7).
Figure 7.
-------
Trouble shooting
[If you get an "Unable to Authenticate" error (Figure 8) - in the My Preferences screen, open
Application, go to the remote IP tab and switch from the windsorsolutions URL to the whatsmyip.org
URL. While there, make sure check box "Always authenticate against NAAS is NOT checked?'.
Click "Save" and return to retry to connect]. For further assistance regarding connection issues,
contact the Node help desk [(nodehelpdesk@epacdx.net); 888-890-1995] or the WQX help desk
STORET@epa.gov 1-800- 424-9067.
Figure 8.
With a successful test connection, the request data page will appear. Notice that all options in the
Things I can do box are now available with a successful test submission (Figure 9).
Figure 9.
Request Troe
0 snow the resJB
Pewntrtyrw*
w (best fa snwl requests)
Sj to -1
O Download law (best fa l»ge rousts)
10
-------
Step 4. Submit a Test XML file
1. In left panel, under Things I Can Do, click "UploadDocuments".
2. In the Upload Documents(s) panel, in the Data Flow field, select WQX from the dropdown
list.
3. If the Transaction ID is prepopulated, delete its contents.
4. click "Add ".
5. A test XML is provided at the following site
http://www.exchangenetwork.net/exchanges/water/wqx.htm. Download the example XML
file and save it to your hard drive.
6. Select your test XML file(s) from your directories (.zip or .xml); the address will appear in
"Selected File(s)" window (Figure 10).
Figure 10.
7. click "Submit".
8. Correct any errors encountered (CDX validation and WQX rules). Users will
receive an email to their NAAS account providing details on either a completed
submission or a failed submission. Please refer to the following links for guidance on
CDX validation
(http://water.epa.gov/grants fundmg/beachgrants/upload/2008 10 01 beaches grants d
atausers 2008 BeachesNotificationCorrections20081001 .pdf) and WQX rules
(http://www.exchangenetwork.net/schema/WQXy2/WQX FCDv2.1 .pdf).
11
-------
9. With a successful test submission, you should get "Document Submission Results" as well
as an email to your associated NAAS account informing you of a successfully submitted file
(Figure 11).
Figure 11
ij7:TB7«.-nCT»n;0r.>Ji..;-ii.:..-)gia^.v^lir;mTl?TO..anigrwinB
Example email from a completed WQX Submission
Your WQX submission, transaction ID [_7d8b81ie-e2cl-4c9b-8cec-52ebef2e458e], has been processed.
The process status is COMPLETED.
The process report, with the document riane [Notify_AS£PA Beach Monitoring Data Q2 FV261d.xal.xal]
and the docu*ent type of [XML] is ready to be downloaded.
Example email from a failed WQX submission
Your HQX submission, transaction ID [_ee7d7b8b-edda-«f&6-a7af-c4dfc3e621Jf], has been processed.
The process status is FAKED.
Th« process report, with the document name [Notify_ASEPA Beach Monitoring Bata Q2 FYJ6ia.x»l.x«l]
a Ad the doeuMflt type of [XML] is ready to be downloaded.
12
-------
Submitting Data Through Production
Once you receive an email confirming the one time successful test submission, you are able to
submit to production. If you signed up for a NAAS account, your test username and password
are most likely your production user name and password,
You can obtain the appropriate production endpoint to submit and receive data by accessing the
exchange network page
http://www.exchangenetworkwiki.com/wiki/index.php/Node Endpoints at U.S. EPA's Central Da
ta Exchange.
If your test username and password does not work, you can obtain a Production password from
888-890-1995; then use the production connection for Node 1.1:
https://cdxnode.epa.gov/cdx/services/NetworkNodePortType VI0.
Version 1.0 of the WQX provides the means for partners to share the following types of data:
The physical conditions in the environment at the time of a site visit;
The chemical and bacteriological make-up of the water sampled;
Chemical analyses offish tissue collected.
Version 2.0 of the WQX expands on WQX vl.O and provides the means for partners to share all vl.O
data types as well as the following:
• Biological Taxon Abundance data, including population census, frequency class, group
summaries, and individual results;
• Toxicity data;
• Habitat Assessment scores and their related metric scores;
• Biological Index scores and their related metric scores.
13
-------
I. To Obtain Data
1. In left pane, in the Things I Can Do box, click on Get Data
2. In the Request pane, in the Data Flow box, use the dropdown option to select WQX (last
one; Figure 12).
3. Example 1: Get Transaction History by Parameters
In the Service box, select WQX.GetTransactionHistoryByParameters_vl.O
a. The Node Client Lite application allows users to specify data by parameters. One or
more parameters can be entered for queries. Parameters can be filtered for begin/end
dates (right click on icons), Transaction ID, or Org ID e.g.: Orgid [WQXTEST]
b. In the Request Type pane at the bottom of the application, users are able to select
their desired output form. Click Submit.
Figure 12.
Data Bow: ,WQX
Connect to node
fi Get Data
« Upb* Documents
ft Download Documents
J
w {best for anal reqjests)
Request Type
0 Show the re
Return Griffons 0 5 to -I
O Downtoad later (best for lags requests)
d. In Z)fl/a Request Results pane the location of your file will be provided. The file name
will be based on: YEAR/MONTH/DAY/HOUR/MINUTE/SECOND, e.g.
C:\2007\08\30\12\49\54.
e. Go to the location on your hard drive where the files were downloaded to open folder
and view files.
f Open transaction file, find most recent at bottom, copy transaction id; e.g.
14fd9e21-c495-4c7f-a309-f3dd522b97bc.
g. See the instructions under HI (To Download Data) for guidance on downloading
files.
14
-------
4. Example 2: Get DomainValuesbyElementName_v.l.O
Note: the "WQX Flow Configuration" document contains the list of domain values
documented in WQX.
a. In the Service box, select WQX. Get DomainValuesbyElementName_v.l.O.
b. Specify element name (optional), e.g. "characteristic" and select submit, see 3.b-3.g.
Additional services users may request or query data from the WQX database
include:
- GetProjectByParameters_v2.0
- GetMonitoringLocationByParameters_v2.0
- GetActivityByParameters_v2.0
- GetActivityGroupByParameters_v2.0
- GetBiologicalHabitatIndexByParameters_v2.0
- GetResultByParameters_v2.0
- GetDomainValueByElementNamejvl.O and_v2.0
- GetTransactionHistoryByParametersjvl.O and_v2.0
• All data submitted via WQX and WQX Web are placed in the STORE! Warehouse
II. To Upload Data
1. In left panel, in the Things I Can Do, box click Upload (Figure 13)
Figure 13.
F»
2. Under the Upload Documents Panel, In the Data Flow box, select WQX from the
dropdown menu.
3. If the transaction ID box is pre-populated, delete its contents.
15
-------
4. Click "Add".
5. Select file(s) from your directories (.zip or .xml); address will appear in
"Selected File(s)" window.
6. Click "Submit", to receive a message titled "Document Submission Results"
a. verify information.
b. block and copy transaction code (use
-------
6. In Document Download pane, below is an example of provided download files
file://C:\20070830124954\BackEndDistribution a6eOeadO-840d-42d5-98b2-4ddc2730e81f.zip
file://C:\20070830124954\Notifv WOX valid file.xml.xml
file://C:\20070830124954\ProcessinaReport.zip
file://C:\20070830124954\Validation Results
file://C:\20070830124954\WOX valid file.xml
7. Files are saved to folder with name based on:
YEAR\MONTH\DAY\HOUR\MINUTE\SECOND, e.g. C:\2007\08\30\12\49\54 (.zip)
8. Review the Processing Report file to see if there is an error message at bottom of the
report. This error message can be used to fix the file.
a. verify information
b. block and copy transaction code (use
-------
Fixing Data Quality Issues /Formatting Data Submissions According To The WQX
Structure
The largest challenge for users of WQX is mapping their water monitoring data to the WQX
schema business rules and domain values. If the file does not pass CDX validation, it does not
continue on to WQX for processing. The WQX processing software checks that files conform to
WQX data element formats, business rules and domain values.
• Key Domain Value Lists:
- Characteristics
• WQX standardizes on the EPA Substance Registry System (SRS) for
chemical names
• WQX will maintain candidate names if a chemical name has not been
vetted through SRS
- Taxa Names
• WQX standardizes on the Integrated Taxonomic Information System
(ITIS) for taxa names and rank
• May need to roll up to lowest ITIS rank name
• Monitoring program working with ITIS system for updates
- Please contact us for new requests
• The CDX node performs validation that the submitted XML is well formed
More detailed information on WQX structure and flow can be found at the following URL
http://www.exchangenetwork.net/schemaAVQX/2AVQX FCDv2.1 .pdf
Once a file is submitted using the production link, users can download the following:
- Original XML file submitted
- CDX validation report
- CDX notification file reporting status as either completed or failed
- WQX processing report summarizing the inserted/updated/deleted data and any
encountered errors
18
------- |