www.epa.gov/research
Technological Tools for Evidence Integration
Shane Thacker, Jennifer Nichols, Ryan Jones, Steven J. Dutton
National Center for Environmental Assessment; Office of Research and Development; US EPA
Shane Thacker I thacker.samuel@epa.gov I 919-541-5159
Evidence Integration
New Tools, continued
New Tools, continued
At the EPA's National Center for Environmental Assessment (NCEA), we work
closely with programs throughout the EPA to integrate web-based and desktop
computer tools into the assessment process, facilitating evidence integration for
science assessment products. By incorporating in-house and third-party tools,
both open source and commercial, activities such as the Integrated Risk
Information System (IRIS) and the Integrated Science Assessments (ISA) seek to
use the best tools for the job, while remaining flexible enough to improve the
evidence integration process.
Current Tools
•	Health and Environmental Research Online (HERO): Literature search,
categorization, acquisition, archiving. Interoperable with HAWC, Distiller, and
SWIFT.
•	Health Assessment Workplace Collaborative (HAWC): Study evaluation,
data extraction, visualization. Interoperable with HERO, BMDS, and Distiller.
•	Benchmark Dose Software (BMDS): Dose-response modelling. Interoperable
with HAWC.
•	Evidence Partners DistillerSR: Literature screening, data extraction.
Interoperable with HERO and HAWC.
•	Sciome SWIFT-Review and SWIFT-Active Sereener: Literature screening,
prioritization, categorization. Interoperable with HERO and HAWC.
Evidence Profile Table, pictured
New Tools
Evidence Profile Table
Part of HAWC, the Evidence Profile Table offers a summary explanation of
evidence integration in a chemical risk assessment. This view creates greater
transparency about the body of evidence by illuminating the rationale behind the
assessment findings.
•	Adaptation of GRADEPro Evidence Profiles
•	Create multiple rows to cover multiple evidence streams
•	Select studies and endpoints added to HAWC
•	Streams break down into scenarios
•	Endpoints are rated within scenarios
•	Confidence judgements build from individual to across-stream
•	Findings summaries add to confidence judgments
*	Profile Streams
Scenario Studies (organized by effect tag)
Factors That Increase Confidence In The Selected Studies
Profile Stream Scenarios
Factors That Decrease Confidence In The Selected Studies
Outcome
Studies
Factors
That
Increase
Confidence
Factors
That
Decrease
Confidence
Confidence
Judgement
and
Summary
of Findings
for
Individual
Outcome
Within-
Stream
Confidence
Judgement
and
Summary
of Findings
Inference
Across
Streams
Across-
Stream
Confidence
Judgement
test animal (Animal Bioassay)
•test inference
this is another
description
®©o
test total
This is a test
explanation
Test scenario
neoplastic
•Initial
submission:
Title of the
study
(Thacker
and Jones
2019)
•Consistency
Relatively Low
•High Risk of Bias
This is a factor
that would
decrease
confidence.
©fflO
Moderate
test outcome
This is another
explanation
eoo
Slight
test judgement
This is a full
explanation
test outcome
Overall Odds
Ratio
• Title of
another
study here
(Thacker
and Nichols
2019)
•Consistency
Relatively Low
•High Risk of Bias
This is another
factor that would
decrease
confidence.
©e®
Robust
test title
test
test summary
within
test summary full
Title of summary
And the full
summary.
Evidence Mapping
Integrated with the Health and Environmental Research Online (HERO) database,
the Evidence Mapping tool allows researchers to create heat maps to visualize and
overlay characteristics (e.g., discipline, exposure, concentration, etc.) of the
reviewed literature, making it easy to visualize the available evidence.
HERO tags used for Evidence Map
Example Evidence Map
a

Health Outcome Category

Mortality
CV
14
Resp
	12
Repro/

Nervous
System
Cancer
Dermal

i
t
US
16
11
2
0
0
0
5
128
Canada
10
1
1
0
0
3
7
48



6
2
5
0
i
5
116
Asia


5
3
8

1
10
178
Other
ii^B ¦»

2
1

0
0
2
128
E
to
8?
US
15
8
18
3
5

5
0
7
106
Canada
5
1
4
1

1
0
:
17
Europe
5
5
11
16
6

2
0
1
46
Asia
10
16
19
25
4

4
1
6
89
Other
3
12
3
15
0


0
4
42
1
US
3
1
3
1
1

1
1
1
13
Canada
0
0
1
0
0
0
0
0
C
1
Europe
0
3
4
2
1
0
0
0
2
12
Asia
0
1
8
2
0
0
0
2
0
13
Other
0
1
2
2
0
4
0
1
1
11
Total
140
227
310
135
27
32
15
10
52
948
•	In HERO, scientists use tags to categorize literature for possible use in chemical
risk assessment projects
•	Using the tool, scientists create crosstabs between sets of tags that code literature
by characteristics
•	The results are color-coded, creating heat maps for easy visualization of the
intersection totals
•	The result is a map showing the amount of possible evidence between
characteristics, such as location and exposure
•	Researchers can layer the characteristics into sets and subsets, adding visual
organization
•	Interoperable with Distiller
Evidence Inventory
The Evidence Inventory tool, hosted within HERO, facilitates data extraction and
portrayal by providing researchers a template to collect and categorize data from the
relevant literature and then create summary tables of the extracted information. The
summary tables are then ready for export into assessment documents, allowing
readers to review the evidence behind the chemical risk assessment.




,00
Reliability/St


Study
Study

Sample

HERO ID
Author
Year
Title ISA Relevant
Relevant
udv Quality
Section
Health Endpoint
Design
Name
Study Population Details
Size
Com
123457 Thacker, S;Joi 2015
123457 Thacker, S; loi 2015
123457 Thacker. 5; Jo>2015
123459 lo
123459 Jo
, R; Dutt 2014
, R; Dun 2014
High Quality
High Quality
High Qu;
High Quality
High Quality
High Quality
High Qi
>i Yes
gasol Yes
gasol Yes
gasol Yes
gasol Yes
High Qu;
High Quality
High Qui
Infl/Ox Stres
Infl/Ox Sti
Infl/Ox Sti
•	Producing a chemical risk assessment document requires extracting and
reformatting the data in cited studies into tables
•	Previously, this was largely a manual task without required standardization
•	This new tool uses spreadsheets formatted for different disciplines to help
standardize data extraction
•	Once the data is extracted, the spreadsheets are transformed into sets of tables
useful in the document production process
Future Development
•	Store data in HERO for repeated use in assessments
•	Develop search and reporting capabilities for extracted data
•	Through text and concept mining tools, automate the first pass at categorization
and tagging
•	Visualize the results of automated categorization in Evidence Maps
•	Create tighter integration between HERO and HAWC
•	Create, improve, and utilize web service APIs for HERO and HAWC to ease
integration with third-party software
•	Allow web-based data entry for Evidence Inventory
•	Integrate Tableau visualization software with HAWC, Distiller, and Evidence
Inventory tools
•	Integrate Evidence Prime's Pupil automated data extraction software with Distiller,
HAWC, and Evidence Inventory tools
•	Investigate possible standards for extracted data formats to ease data migrations
•	Investigate and implement tools for automated table and graph data extraction
•	Work on ontologies for data extraction to make the data more easily searchable
•	Employ agile development processes to test and incorporate new and useful tools
into the assessment process
U.S. Environmental Protection Agency
Office of Research and Development
Disclaimer: The views expressed in this poster are those of the authors and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency.
Any mention of trade names, products, or services does not imply an endorsement by the U.S. Government or the U.S. Environmental Protection Agency (EPA). The EPA
does not endorse any commercial products, services, or enterprises.

-------