WQX User Call
January 26, 2023
There were approximately 43 participants on the call.
Returning Attendees:
Bill Kramer (EPA)
Adam Griggs (EPA);
Shari Kondrup Brick Utilities
Shay Torres (Contractor to EPA)
Valarie Fields (Contractor tot EPA)
Jolene McQuillan (Montana DEQ)
Chris Neumiller (WA Ecology)
Steven Vu (EPA R7)
Cathy Anderson (TCEQ)
Candice Hopkins (USGS)
Cristina Mullin (EPA)
Elizabeth Smith (EPA R4)
Dawn Hintz (SRBC)
Jarma Bennett (CA State Water Resources Control Board)
Amber Bethe (AK DEC)
Elise Hinman (EPAORISE)
Adam Fisher (EPA R2)
Jesse Boorman-Padgett (EPA HQ)
Lee Stanish (USGS)
Kayla Gower (WQX helpdesk contractor)
Rick Cooke (Gold Systems)
First-time Attendees:
Lynette Guevara (NMED) - lvnette.guevara@env.nm.gov
Ben Frans (WA Ecology) benf461@ECY.WA.GOV
.Agenda:
1. WQX Project Status Report (25 mins)
o WQX/WQXWeb performance/updates (Kevin)
o Characteristic Group Refinement: Discussion and user feedback
-------
o WQX Monthly User Tip (Adam) - Intro to using WQP web services in online data
applications
o WQX questions/comments
2. WQP Status Report (10 mins)
o WQP general status report (Candice)
i. Updating web service guidance for WQP
ii.
o WQP questions/comments
3. Open Discussion
o Other topics/questions
Notes:
Welcome. We are trying out a new notes system during these meetings that is more in keeping
with the practices of our other External User groups. Feel free to add your name to the
attendance list so that we can have a better record of who is attending these meetings and
receiving these updates.
1. WQX Project Status Report
a. WQX/WQXWeb performance/updates
The spring release for WQX Web will bundle application and performance bug
fixes for the current azure cloud deployment. There are several new features
being deployed. Our current QAQC service reports, previously generated only
during submission, are being made accessible via WQX Web via the Review
menu. The QAQC reports will also now be available to produce on any previously
submitted data. The QAQC reports are available for both Locations, and Results.
There are also new reports available that will produce ALL WQX 3.0 schema
entities. These will be made available in a queued download. These will be
available for all schema levels, including: Locations, Projects, Activities, and
Results, etc. We do not currently have a report for Project Location Weighting.
We will also be releasing REST style Rest URLs for downloading reports based on
the Transaction ID. The REST URL will not require CDX authentication, so it will be
publicly accessible via the REST URL.
• Q Steven R7 - Are the QAQC reports something that are already available?
What will they look like? Answer: They are spreadsheet reports that will
-------
include the data in a summary format, with indications of whether
locations map correctly to the submitted assignments of county, state,
country. For results, we compare typical ranges, result special characters,
QAPP information, range checks (upper and lower thresholds), units,
methods, units, fractions, and speciation.
• Q Chris (ECY) - How do these requests work in concert with the service?
Answer: The QAQC reports are automatically generated after you submit.
You can download these reports after submission.
b. Characteristic Group Refinement: Discussion and user feedback
Background: The WQX Team is currently exploring improvements and
refinements to the Characteristic Group options and domain service. The
Characteristic Groups are not a WQX submission field. Users only submit the
Characteristic Name. We assign all of our registered Characteristics to Groups
that are added to the data when passed to the Water Quality Portal. These
groups help assist users in data discovery. Each CharacteristicName is currently
assigned to a single group in a 1:1 relationship. Example groups include, PFAS,
biological, metals, nutrients, etc.
Current issues:
1. Many Characteristics are currently Not Assigned.
2. There are incorrect assignments in the table.
3. The table currently only supports 1:1 relationships. All Characteristics can
only be in one Group.
Example Use Case: A Group for rapid retrieval of HAB Characteristics
Harmful Algae Characteristics tend to stretch across the entirety of our sample
schema, including chemical measures, indices, biological counts, tissue
measurements, etc. How do we best serve up Groups within this context? We
have found it difficult to stay within a 1:1 assignment as many measures may
potentially apply to more than one group.
Goal: Try to implement a l:many assignment without causing Portal disruption.
After some thought, our current proposed solution may include concatenated
assignments where you could assign more than 1 group to a characteristic. We
-------
wanted to get any feedback the group may have on moving toward concatenated
group assignments. Example: 1:111:211:31 Biological | Algae, Cyanos, Periphyton,
and Plants| Habitat
1. Cathy Anderson: In my experience, you can run into issues with the
system processing the request if it is a large group, especially if you throw
in a date range, and you can run into issues with processing the data on
your local workspace. Would caution us on implementing a l:many.
2. Chris Neumiller: These are useful, and this is a good idea, but suggest
designing it in a way so we can move it to a table in the future easily.
From chat: "Just make it so that you can eventually put it in a relationship
table".
C. Intro to using WQP web services in online data applications
Adam: Live demo of using WQP web data services in various applications (Arc
Online, Excel, googlesheets/LookerStudio
c. This is for the stations, results, and summary services (steps below)
i. Using WQP Ul, generate your query and copy the station web service
ii. Paste the station web service into Arc Online within the map (easier but
not limited) or hosted feature service, larger queries should use hosted
feature service
iii. When adding within the map, choose the file format to "A CSV file" and
change zip=yes to zip=no in URL
iv. New map viewer for Arc Online can support JSON and csv using both
methods for bringing in data, but that is not true for past versions
v. When adding layers, you can change the field type (string, numeric, etc)
vi. Assign correct fields as the lat and long measures
vii. Add data layer to map
1. Now you can see all locations where data is available for your
query.
Station Service:
Example: All locations with records for "Micropterus dolomieu" the smallmouth bass.
httpsi//www.waterqualitydata.us/data/Station/search?subjectTaxonomicName=M
-------
icropterus %2 Odolomieu&mimeType=csv&zip=no&providers=NWIS&providers=STEWARD
S&providers=STORET
Adding the WQP Services to Arc Online
There are many options when it comes to adding WQP web services to the Arc Online environment. This
guidance is just an introduction for the most easily accessible functions and services. Generally, adding
data directly to a map by either csv or geojson will be limited compared to a hosted feature service that
establishes a connection to the WQP service. Also, there are differences between the Classic Map Viewer
and the New Map Viewer, in that there is generally more functionality and compatibility in the New Map
Viewer.
A B C . D E F
Arc Online Classic
New Ma
p Viewer
Add in Map (Reference)
Hosted
Add in Map (Reference)
Hosted
Station service
CSV
Works
https://www. waterq u a 1 ityd at a. u s/d at a/St a t
ion/search?
cou ntrycod e=US& statecod e=US%3 A24&cou
ntycod e=US963 A24963AQ3 3 & m i m eTy pe=csv
&zip=no&providers=NWIS&providers=STEW
ARDS&providers=STORET
No option from map or Content
window
Works
https://www.waterqualitydata.us/data/Stat
ion/search?
cou ntrycod e=US&statecode=US%3A24&cou
ntycod e=US%3A24*3A033&mimeType=csv
&zi p=no& provid ers=NWIS& provid ers=STEW
ARDS& provid ers=STORET
Works
https://www.waterqualitydata.u5/data/Stat
ion/search?
countrycode=US&statecode=US963A24&cou
ntycode=US%3A24963A033&mimeType=csv
& zi p=no& provide rs=NWIS& providers=STEW
ARDS&providers=STORET
geojson
No option - Geojson not listed as
filetype in the add from web
option
Works
https://www.waterqualitydata.us/data/Stat
ion/search?
cou ntrycod e=US&statecode=US%3A24&cou
ntycod e=US%3A24%3 A033& m i m eTy pe=geoj
son &zi p=no& provid ers=NWIS& provid ers=ST
EWARDS& provid ers=STORET
Works
https://www.waterqualitydata.us/data/Stat
ion/search?
cou ntrycod e=US&statecode=US%3A24&cou
ntycod e=US%3A24%3A033& m i m eType=geoj
son&zip=no&providers=NWIS&providers=ST
EWARDS& provid ers=STORET
Works
https://www.waterqualitydata.u5/data/Stat
ion/search?
countrycode=US&statecode=US%3A24&cou
ntycod e=US%3A24%3A033& m i m eTy pe=geoj
son & zi p=no& provid e rs=NW IS& provid e rs=ST
EWARDS& provid e rs=STORET
Adding WQP Stations data from the Web (Classic Map viewer)
In your ArcGIS Online account, Open a Classic Map Viewer. On the leftOhand-side, select the "Add"
dropdown and select "Add Layer from Web". Select "A CSV File" from the file type dropdown and paste in
the WQP Station Query URL that was generated by your search. Edit the URL zip instructions to read
"mimetype=csv&zip=no" and click the "ADD LAYER" button. A pop-up will ask you to identify the latitude
and longitude fields in the data. Scroll down the data fields to correctly identify "LatitudeMeasure" and
"LongitudeMeasure" as the respective fields. Select "ADD LAYER" again and the stations should appear in
the map.
*NOTE - This call will be too big to import directly into the classic map. QUeries of this size or larger may
only be supported through a hosted feature service. Currently, hosting CSVs is only made possible
through the New Map Viewer add layer button.
-------
Calgary
Reglna
Winnipeg
Adding WOP Stations data from the Web (New Map Viewer)
In your ArcGIS Online account, Open the Map Viewer. Under the layers tab, open the add dropdown and
select "Add layer from URL". Paste the Station query you copied above into the URL field. You may add
the layer as either a csv or GeoJSON but both should be imported as a non-zipped file. Edit the URL to
read "mimeType=geojson&zip=no". Next, select the layer type as GeoJSON, then click next. You will be
given the option of adding the layer to the map, or hosting the web layer in your content. For best
results, larger queries should use the hosted feature layer option. Select "Add to Map". The stations you
queried should appear on the map.
Result Service:
This time, let's import a Results service query into Excel
Example: All nutrient data collected from Bedford County, Tennessee
WQP Query URL:
https://www.waterqualitydata.us/#statecode=US%3A47&countycode=US%3A47%3A003&characteristicT
ype=Nutrient&mimeType=csv&dataProfile=narrowResult&providers=NWIS&providers=STEWARDS&provi
ders=STORET
-------
WQP Results Service URL:
https://www.wateraualitvdata.us/data/Result/search?statecode=US%3A47&countvcode=US%3A47%3A0
Q3&characteristicTvpe=Nutrient&mimeTvpe=csv&zip=ves&dataPrQfile=narrowResuit&providers=NWIS&
providers=STEWARDS&providers=STORET
In Excel, we navigate to the Data tab of the ribbon and select "From Web" within the "Get & Transform
Data" section.
File
a
Get From
Data " Tesct/CS1
sert Draw Page Layout
Do
Lq
rom Table/ Recent Existing
Range Sources Connections
Transform Data
Formulas ^^^^1 Review
n Queries & Connections
i
erties
Refresh
All - [I, Edit Links
Queries & Connections
View
n
u
Help
Acrobat
Sort
rate, 'Bl""
^ Advi
Sort Bt Filter
In the pop-up, simply paste in the Results URL copied from the portal and change the "zip=yes" to
"zip=no" and hit OK.
r
>
X
From Web
® Basic O Advanced
URL
|=Nutrient&mimeType=csv&zip=no&dataProfile=narrowResult&providers=tf
OK
Cancel
j
The data will be imported and parsed into the spreadsheet, ready for analysis.
A B C D | E F
brganizationldentifier D OrganizationFormalName
Q Activityldentifier
Q ActivityStartDate D ActivitYStartTime/Time O ActivityStartTime/TimeZoneCode O Monitorirl
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700504
3/27/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700504
3/27/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700511
2/1/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700511
2/1/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700513
3/27/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700513
3/27/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700502
2/1/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700502
2/1/1967]
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96600332
9/29/1966 3:15:00 PM CST
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96600332
9/29/1966 3:15:00 PM CST
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700501
11/17/1966
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700501
11/17/1966
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700512
2/13/1967
USGS-035!
USGS-TN
USGS Tennessee Water Science Center
nwistn.01.96700512
2/13/1967
USGS-035!
iJSfi-S-TN USfiSTpnnp<;<;pp Watpr Scipnrp Cpntpr nwistn.01.196700515 5/11/1967 USfiS-OSV
Summary Service:
This time, let's test the services in Google products.
-------
You can find the documentation for WQ.P web services here:
https://www.waterauaiitvdata.us/webservices documentation/
Example: Summary service request to retrieve total counts from the most
recent 5 years of Nutrient data from sites in Benton County, Arkansas in
GeoJSON format, zipped:
https •,/ J www. waterqualitydata . us/data/summary/monitoringLocation/
search?countycode=USS3A05%3A0 07 & characteristicType=:Nut.rient &zip=
yes&dataProfile=summaryMonitoringLocation&summaryYears=5&mimeTyp
e=:geo j son
In googlesheets, we click into the first cell and insert a simple function =IMPORTDATA(). The function
consists of the URL, the delimiter used, and a country code. We paste in the generated URL and switch
the zip=yes to zip=no. See below:
=IMPORTDATA("https://www.waterqualitydata.us/data/summary/monitoringLocati
on/search?countycode=US%3A05%3A0 07ScharacteristicType=Nutrient&mimeType=cs
v & zip=noSdataProfile=periodOfRecordssummaryYears=5", ",", "en US")
¦k Exampie_WQP_Summary_Service ~ O <2>
¦¦ File Edit View Insert Format Data Tools Extensions Help Last edit was 2 hours ago
T S % .0_ .00 123' Default(ArL. ^ 10 - B J -fr A ^ S Es ' i ' V' C-: [3 @ *
AI ~ j =IfWRTDATA( "https: //wwv^ waterqualitydata. us/data/summary/nK>nitoringLocation/search?CQuntycode=US%3A05%3A0078characteristicType=Nut
aIb c d e f g h i j
1 =II^P0RTDATA(" https: //www. waterqualitydata. us/data/ summary/nK>nitorir>gLocation/search?countycode=iJ5%3AS5%3A0O7&characteristicType=Nutrient&niimeType=c
2
NWIS
USGS-0704905<
2019 Nutrient
Ammonia and ar
4
8
USGS-AR
USGS Arkansas W«
IjlI
NWIS
USGS-0704905C
2019 Nutrient
Inorganic nitroge
4
4
USGS-AR
USGS Arkansas Wc
4
NWIS
USGS-0704905(
2019 Nutrient
Kjeldahl nitrogen
4
4
USGS-AR
USGS Arkansas We
5
NWIS
USGS-0704905(
2019 Nutrient
Nitrate
4
8
USGS-AR
USGS Arkansas We
6
NWIS
USGS-0704905(
2019 Nutrient
Nitrite
4
8
USGS-AR
USGS Arkansas We
7
NWIS
USGS-0704905(
2019 Nutrient
Nitrogen, mixed
4
4
USGS-AR
USGS Arkansas We
8
NWIS
USGS-0704905(
2019 Nutrient
Organic Nitroger
4
4
USGS-AR
USGS Arkansas We
9
NWIS
USGS-0704905(
2019 Nutrient
Orthophosphate
4
8
USGS-AR
USGS Arkansas We
in
MWI.Q
Wntrifnt
Phrtrnhnrii':
A
A
I AR
I A rim rime Wr
This data is now available for use in a variety of online data visualization applications.
-------
Provider
OrgantzationFor.-
Record Count -*
1. NWIS
USGS Arkansas Wate...
242
Provider
Z STORET
Arkansas Water Reso„
69
# Stations
16
Record Count
385
1-4/4 < >
CharacteristicName
OrganizationFormaiName
7"
Anderson
Pinevllle
^^^Gravett
m
Gentry
w
Slloam Springs
Spnngdale™
Eureka
. w
Befryvitl
r
"
§
Kmdsville A'aba
+
Hunlsville
ResultCount
12 • «
Prairie Grove
Keyboard shortcuts Map data S2Q23 Google Terms of Use
OrganizatfonFormalName
£ USGS Oklahoma Water Science Center < )
ft Arkansas Wat® Resources Center
lllllllll.
# Nitrate
# Kieldahl nitrogen
# Nitrite
# Phosphorus
# Organic Nitrogen
# Nitrogen, mixed
forms (NH3). (NHi),
organic. (N02) and
(N03)
# Orthophesphate
# Ammonia and
# Inorganic nitrogen
(nitrate and nitrite)
# others
Data Lass Updated ' 26/2023 9 4-1:59 AM Privacy Pafrc.
4. WQP Status Report (10 mins) (Candice and Lee)
We will be adding new documentation for the web services soon, USGS is updating some
of our reporting limit values and if you are a frequent user of USGS data, you may see
that change as we are able to report additional reporting values.
The new dev team is not actively working on WQP at this time, it has been working on
NEMI.
Any questions for the WQP Team?
Q - Mary Becker - Are there plans to migrate the old BioData database to WQP. Nope,
there are not any current plans to update or revive the BioData database.
5. Wrap-up
• We will look into potentially putting together a web services training
-------
• Google Looker Studio is a free-to-use online data visualization tool, Adam wrote
guidance in the notes above on how to bring in data via all the applications he
demo'd today
• HMW is also a demonstration of use of web services, we have WQP Monitoring
Data pages with summary charts and tables for each station
• On the next call, we will get a presentation from the Massachusetts Bays
National Estuary Partnership on their recent development of an R-Shiny based
submission tool and shared WQXWeb import configuration strategy to support
community science submissions to WQX web.
The next WQX/WQP User Call will be on the 4th Tuesday of February (Feb 23rd) at 12pm EST.
------- |