tnvironmental Protection
Agency
Office of Enforcement
and Compliance Assurance
(2223A)
EPA 305-R-99-003
December 1999
Sector Facility Indexing Project
Evaluation: December 1999
-------
-------
Sector Facility Indexing Project Evaluation
December 10,1999
prepared for:
Office of Enforcement and Compliance
U.S. Environmental Protection Agency
(2223A)
401 M St., SW
Washington, DC 20460
EPA Contract No. 68-W6-0021
-------
TABLE OF CONTENTS
Executive Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Background. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
What is SFIP? [[[ 6
Goals of SFIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Innovative Features of SFIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Organization of Evaluation Results. . . . . . . . . . . . . . . . . . . . . . . . . . . .. ................. 10
Sources of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Evaluation Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Goal 1: Public Access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Awareness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Extent of Awareness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Effects of Awareness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Customer Satisfaction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Are SFIP data useful? .......................................20
Are SFIP data understandable? ................................ 23
Are SFIP data easy to find and use? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Information Use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Use by Environmental Organizations: ..........................27
Use by EP A ............................................... 28
Use by Industry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Goal 2: Improve Multimedia Facility Profiling and Sector-Based Analysis. . . . . . . . . .30
Advancing Data Integration Knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Assessing Data Quality in Federal Data Systems. . . . . . . . . . . . . . . . . . . . . . . . 35
Data Quality Assurance Review and Assessment of Data Quality. . . . .36
Quality Assurance Procedure for Updating SFIP Data. . . . . . . . . . . . . . 37
Future Directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Additional Sectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Alternative/Additional Functionality of the SFIP Website. . . . . . . . . . . . . . . . .42
-------
Appendices
Appendix 1: Website Use Statistics Reports
Appendix 2: Sample Discussion Questions from SFIP Focus Groups
Appendix 3: SFIP Press Release, May 1, 1998
December 1999
-------
EXECUTIVE SUMMARY
The Environmental Protection Agency (EP A) collects and maintains environmental performance
information on hundreds of thousands of facilities and operations subject to environmental
regulations in the United States. Facility-level information is collected under many different
regulatory programs and is typically stored in separate federal data systems. The Sector Facility
Indexing Project (SFIP) was initiated in 1995 as a pilot to test several new policy and program
alternatives. In particular, the project sought to demonstrate that there is value in organizing
facility-level information along industry sectors. In addition, SFIP demonstrated that the
information, while admittedly complex, can be summarized in a way that will allow the general
public, as well as environmental organizations and industry, to better understand the
environmental performance of facilities. Following a development stage in which many types of
stakeholders were consulted, EPA launched the SFIP website on May 1, 1998. Website users are
able, for the first time, to conduct their own analyses on-line and look up facility-level
information that is both complete and current.
This evaluation of SFIP, undertaken at the end ofthe project's first full year of operation, reveals
several findings that are important for the many individuals and organizations involved in
developing environmental policy and programs in the U.S. and abroad. The most significant
finding of the evaluation is that simplifying facility-level records and providing easy access to the
data has created value for environmental managers at all levels of government, the public and
even the regulated community. Users have been especially appreciative of the reliable sector
universes and permit linkages for individual facilities that are readily available on SFIP's
website.
The evaluation shows that SFIP is also valuable as a laboratory for examining environmental data
management and analysis procedures. Most importantly, SFIP has proved that it is feasible to
link all records belonging to a facility, a key enhancement for sector-level analysis. In addition,
the project's continuing commitment to data quality review has resulted in the identification of
several opportunities for further improvement, including inconsistent interpretation of coding
guidelines across Regions/states (e.g., what constitutes an administrative order) and errors in data
processing (such as duplicate records of enforcement actions). All ofthe identified data
problems have been or are now being addressed. Finally, SFIP continues to provide important
baseline information that the Agency is using in its current work on consolidating facility
reporting, improving integration of facility records and streamlining the flow of data from state
and Regional data systems to the federal level.
SFIP has provided EP A with an important new tool to support the re-orientation of the Agency's
numerous data analyses along industry sector lines. Users are eager to see SFIP continue this
contribution by expanding to additional sectors and adding more data. Throughout the first year
of the website's availability and during the evaluation process, users have recommended
numerous sectors for SFIP's expansion. The list of suggested sectors that are both feasible to
1
December 1999
-------
include and that are Agency priorities, includes, but is not limited to: metal services, electric
utilities, paper manufacture, and federal facilities.
This report documents the extent to which SFIP met its original goals, how it has affected a
variety of environmental management activities for the five SFIP industry sectors, and
stakeholders' recommendations for enhancements and expansion of SFIP.
2
December 1999
-------
OVERVIEW
The U.S. Environmental Protection Agency's (EPA's) Office of Compliance (OC) initiated the
Sector Facility Indexing Project (SFIP) in 1995 to empower citizens and communities to protect
public health and the environment by providing the public with high-quality environmental
information for five industry sectors. Expanded public access to environmental information has
been mandated in recent right-to-know legislation and was identified in EP A's 1997 Strategic
Plan as one of the Agency's ten strategic long-term goals.
SFIP is a pilot project intended to make facility-level compliance information readily available to
the public in one location on the Internet. On May 1, 1998, EP A Administrator Carol Browner
publicly launched the project's website (www.epa.gov/oecalsfi). The pilot effort required
improved data integration among the various federal data systems and improved formatting of
compliance information to reduce its complexity and facilitate its use. The project has two
primary goals:
.
Provide greater public access to accurate compliance and facility-level information; and
Improve multimedia facility profiling and sector-based analysis.
.
Because of the project's innovative nature, the Agency committed itself to extensive monitoring
ofSFIP's impacts and a thorough evaluation of the project at the conclusion of the first year of
operation. This evaluation report draws on many sources of information:
.
Calls to SFIP's User Hotline, and emailed and written comments;
Website use statistics;
Feedback on the project from RegionaVState demonstrations of the website;
Discussion groups held with EP A staff, representatives from environmental
organizations, facilities included in SFIP, state environmental agencies, and trade
associations.
.
.
.
Evaluation Results
This evaluation shows that SFIP has been overwhelmingly successful in achieving its two
primary goals: providing public access to compliance information and improving multimedia
facility profiling and sector-based analysis. The principal findings of the evaluation include:
1. SFIP is being used extensively for research by thousands of users and is considered a
valuable tool.
The website allows users to review the records of individual facilities, to conduct on-line
analysis, and to download information for off-line analysis. Over 66,000 user sessions and
404,000 hits have been logged on SFIP's website since its launch. Since it has been publicly
available, the site has hosted an average of 4, 183 user sessions per month, approximately
10,000 sessions per sector in a year. Individual citizens and grassroots groups have used the
site to profile nearby facilities. Public interest groups have conducted analyses of sectors and
individual facilities within a given sector. Industry has used data from the site to publicly
showcase their compliance history in press releases, and to review their own compliance
3
December 1999
-------
records and that of competitors. EP A has used SFIP to conduct a variety of analyses,
including comparative analyses ofEPA program efforts (e.g., inspection rates).
2. SFIP has met the challenge of summarizing complex compliance and pollutant release
information from multiple statutory programs.
SFIP increased the consistency of definitions and carefully matched the informatio~ extracted
from five different programs (Clean Air Act, Clean Water Act, Resourc~ Con.servatlOn and
Recovery Act, Toxics Release Inventory, and Emergency Response NotIficatIOn System).
This approach allowed the information to be summarized at the facility level. SFIP has also
made possible several new types of analyses that summarize compliance and pollutant release
information. Examples include:
.
Analyses prepared from SFIP's downloadable spreadsheets have been used by
EP A Regional offices to compare their programmatic efforts and the performance
of facilities within the Region to national averages.
On-line queries were used to identify facilities with unusually high or low
frequencies of reported pollutant spills -- for increased compliance efforts and for
verification of other spill reporting (e.g., under EPCRA Section 313).
OECA's Sector Strategies for Petroleum Refining, Nonferrous Metals, and Iron
and Steel have used SFIP to profile existing compliance problems and to establish
benchmarks for ongoing compliance trend analysis.
.
.
3. There is a high level of confidence in SFIP's data because ofOECA's commitment to
providing and ensuring quality data. An initial Data Quality Assurance Review by states,
EP A Regions and profiled facilities prior to the SFIP website launch showed EP A's
compliance data to be of high quality. An emaillink on the website and a project Hotline
allow facilities to submit data correction requests at any time. Users regard SFIP's data as
"quality-checked" and consider this one of the project's best features.
4. SFIP brought to light a number of areas for improved data maintenance.
SFIP has identified opportunities for improving the processing of compliance data as it is
compiled into EPA's data systems and into OECA's Integrated Data for Enforcement
Analysis (IDEA) system. SFIP's identification of issues such as the need for more consistent
entry of retum-to-compliance dates for RCRA violations has improved EP A data quality.
SFIP has also demonstrated the need to simplify the procedures facilities must follow to
correct the data maintained in AFS, PCS, and RCRlS, the regulatory "program" databases
that maintain data presented in SFIP. Industry representatives have suggested that EP A
appoint an "ombudsman" for facility data correction requests, who would help guide facilities
through this process. Lessons learned in the course of SFIP's development are now being
used in the Agency's data system modernization work, for example, by the Agency's Facility
Identification Initiative, which is creating a central federal registry system where information
common to multiple permits can be more easily entered and maintained.
4
December 1999
-------
5. Problems anticipated by the states prior to the launch of SFIP did not occur or have been
quickly resolved.
Success in this area was due to quality assurance and quality control efforts, and EP A's
willingness to work with facilities to make corrections when appropriate. For example, some
states initially expressed concern that their resources would be taxed by information and
interpretation requests from the public. However, states have not reported any such problems
since the website's launch. States which participated in a 1999 discussion group verified that
the SFIP launch did not increase their workload associated with responding to public
mqumes.
6. Users have strongly urged EP A to extend SFIP to cover additional sectors in order to capture
the potential value demonstrated by the project's successful pilot effort. Requests have been
received from all stakeholders for access to data for more industry sectors in addition to the
five currently included in SFIP
Future Directions
In evaluating SFIP, the Agency looked to the users of the information for feedback. Many users
have provided valuable insight on how to make the site even more accessible and user friendly,
and how they would like the SFIP to move forward. Issues that some users have asked EP A to
consider in the SFIP planning process include:
.
Adding new sectors, including but not limited to: metal services, electric utilities, paper
manufacture, and federal facilities.
.
Adding new data elements to information already provided, including, for example: pollutant
releases and waste management information from CW A, CAA and RCRA; OSHA
inspections and more details regarding permits (e.g., expiration dates and emissions limits).
These additions will enhance the whole-facility profile provided by SFIP of each facility's
operations, environmental performance and the geographic area in which they are located.
.
Providing users the ability to do geographic mapping and/or to conduct cross-sector analysis
in a geographic area. This might include a listing of facilities within three miles of a location
which are known to release pollutants, or a profile of facilities within a municipality.
.
Extending compliance information beyond two years and giving users the ability to conduct
time series analyses. Physical, organizational and operational improvements (e.g., new
treatment plant) at a facility can take several months or several years to manifest themselves
in changed performance. Extending the time period and the addition of time series analysis
would allow users to distinguish changes in performance at the facility and sector levels over
a longer time period.
5
December 1999
-------
BACKGROUND
As part ofthe Environmental Protection Agency's (EPA's) reinvention efforts to find new
approaches to improve environmental compliance and performance, as well as to encourage
environmental stewardship, EPA is undertaking an increasing number of project-specific
evaluations which take stock of innovative activities. Such evaluations enable the Agency to
assess a project's effectiveness and to plan for continual improvement.
The purpose of this report is to evaluate the Office of Compliance's (Oe's) Sector Facility
Indexing Project (SFIP) and summarize users' recommendations for SFIP's future development.
EP A believes that it is critical to inform and involve all those interested in projects like SFIP, to
solicit stakeholder input, and to improve environmental performance while exploring better ways
to achieve EPA's goals.
What is SFIP?
The Sector Facility Indexing Project has two major functions: it is both a community-right-to-
know project and a data integration pilot. The SFIP website gives the public access to
environmental performance data for five industry sectors. Whole-facility environmental profiles
presented on the website are the result of SFIP's pilot effort to bring together, or "integrate"
facility data' from a wide range of sources. In keeping with community right-to-know principles,
the Agency is committed to achieving cleaner, cheaper, and smarter results through efforts such
as SFIP which expand public access to environmental information and thereby increase public
involvement in environmental protection.
SFIP's website (www.epa.gov/oeca/sfi) profiles approximately 650 individual facilities in five
industry sectors: automobile assembly, pulp manufacturing, petroleum refining, iron and steel
production, and the primary smelting and refining of nonferrous metals (aluminum, copper, lead,
and zinc). For each facility included, SFIP provides environmental data such as the number of
inspections, compliance with federal regulations, enforcement actions taken, chemical releases,
and pollutant spills. SFIP also includes background information on the location and production
capacity of each facility, as well as information on the population ofthe surrounding area. Web
pages describing the data indicators presented and summarizing the project's history provide
additional context to enhance users' ability to understand and use SFIP's data. Importantly, SFIP
has continuously sought stakeholder input on all aspects of the project, including the facility data.
EPA has established a dedicated hotline to support SFIP users, and the project's website allows
users to email comments directly to the Agency.
The SFIP website is unique in that it brings together environmental and other information from a
number of data systems in one comprehensive database. Users can access facility level
information as well as whole-sector profiles. Three standard presentations of facility data are
available, while a "Custom Search" option allows users to design their own reports by selecting
up to 6 of 29 possible searchable indicators at a time. Information about facilities and/or sectors
may also be obtained for all indexed facilities in a given city, state, or EP A Region.
6
December 1999
-------
Goals of SFIP
This evaluation considers SFIP's success in meeting its two major goals:
.
Provide greater public access to accurate compliance and facility-level information; and
.
Improve multimedia facility profiling and sector-based analysis
Expanding public access to environmental information has been one of EP A's top priorities since
it was first mandated by "right-to-know" legislation, including Title III of the Emergency
Planning and Community Right-to-Know Act (EPCRA) of 1986, which established the Toxics
Release Inventory (TRI). EP A's own 1997 Strategic Plan designated increased public access to
environmental data as one of ten strategic long term Agency goals.
The 1997 Strategic Plan specifies that EP A provide communities with easy access to information
about the state of their local environment, and provide the public with tools to protect their
families and their communities. EP A has identified several strategies to meet these goals,
including, but not limited to: increasing the quality and quantity of general environmental
education, outreach and data availability programs; providing information tools which empower
communities to make decisions; making all non-confidential information and data at EP A
available to the public; improving electronic access to information by significantly expanding the
type and amount of information available on the Internet; and ensuring citizen access to the
compliance and enforcement records of regulated facilities so that communities can easily
monitor the extent of facility compliance with environmental laws and permit conditions.
SFIP was developed as part of these "right-to-know" efforts, as a primary pathway through which
facility-level environmental information can be disseminated to the public. EP A anticipates that
SFIP will further the dialogue among regulated businesses, their surrounding communities, and
government agencies; assist the public in examining and comparing records of individual
facilities in nearby communities; and assist businesses and corporations in tracking their own
performance. Finally, EPA expects that SFIP can be a useful planning and analytical tool for all
levels of government.
To enhance the usefulness of the multimedia information to be made public, EPA sought to
create whole-facility and whole-sector environmental profiles; establish reliable rosters or
"universes" of all facilities within a given industry sector; and standardize the information to the
extent possible. Currently, EP A maintains data collected under the Clean Air Act (CAA), Clean
Water Act (CW A), Resource Conservation and Recovery Act (RCRA) and Toxics Release
Inventory (TRI) data in several separate databases. In addition, each program database uses a
different facility identifier, and may track the same facility under different names or even
different addresses. For these reasons, it is difficult for government and public data users to
reliably identify all facilities within a given industry sector (the sector "universe") and to obtain a
complete environmental profile for an individual facility.
7
December 1999
-------
EP A initiated SFIP in early 1995 by first researching which facilities fall into each industry
category and then performing extensive, iterative, automated and manual procedures to "link"
permits tracked in the program databases to these facilities. The initial sectors were chosen
based on the following criteria:
.
the sectors fell within manufacturing SIC codes (SIC 20-39),
the sectors had a small number of facilities (under 1,000),
facilities had high relative pollutant output per facility,
facilities had relatively similar processes and products, and
the sectors had several comparative sources of facility data.
.
.
.
.
This methodology allowed EP A to make use of existing pollutant release data under the Toxics
Release Inventory (TRI); initially constrain analysis to sectors of manageable size for this pilot
effort; and allow a certain degree of facility comparison.
EP A then spent three years assembling environmental profiles for the over 600 facilities in
SFIP's industry sectors and designing and developing a publicly accessible website. As part of
the planning process, EP A solicited input from multiple "stakeholder" groups, including offices
within EP A, state governments, environmental organizations, individual facilities SFIP planned
to profile, and the trade associations for those industry sectors. The Agency also undertook a
comprehensive data quality review to prepare the data for its public release on May 1, 1998.
Innovative Features of SFIP
.
SFIP integrates data from numerous EP A databases and external sources to provide one
comprehensive database of facility-level profiles for the complete "universe" of facilities
in five industry sectors. In the past, users had a difficult time accessing this information
because it was dispersed across several different databases which are not directly
accessible by the public.
.
SFIP extracts and displays key data elements ("indicators") from among the many layers
of complex data maintained by EP A in various hard-to-reach databases in order to
provide accessible and useful environmental facility profiles. Customized data reports
available on the website present both "raw" and aggregated data for each facility,
allowing users to perform comparative analyses as well as more thorough inquiries into
individual facility records. The ability to generate such customized reports is unique to
SFIP.
.
EP A has made continual efforts to ensure that SFIP provides the highest possible quality
of data to the public. In response to stakeholder input elicited during the project's
development, EP A implemented an unprecedented initial quality review of SFIP data.
State agencies and EP A regional offices reviewed data from SFIP's environmental
profiles between November 1996 and February 1997. In addition, in August 1997 EP A
provided each facility that the Agency proposed to include in SFIP with a copy of its
environmental records. Concurrently with the direct facility mailing, EP A established a
hotline to answer questions concerning facility data or the review in general. While the
8
December 1999
-------
data quality review was voluntary and participation was solely at the discretion of the
individual facilities, approximately two-thirds of the profiled facilities chose to submit
comments. Corrections proposed by facilities were then forwarded to the states and EP A
Regions for resolution. Since the website's launch, facilities have been encouraged to
continue to review their updated data reports and to submit comments at any time via the
SFIP Hotline, the website's emaillink, or by mail. In addition, since the launch,
discussion groups have been convened with facilities, environmental groups,
representatives from state environmental agencies, and EP A staff. These discussion
groups provided a further opportunity to obtain feedback about the accuracy of
information presented on the website.
9
December 1999
-------
ORGANIZATION OF EVALUATION RESULTS
In assessing SFIP's success, this evaluation first considers the project's public access goal and
then reviews the its data integration effort. Additional comments and recommendations for
SFIP's future development are then discussed.
Results of the evaluation are presented using the following structure:
* Goal 1: Provide greater public access to compliance and facility-level
information
./ Awareness: .
What is the extent of pubic awareness of SFIP? What are the effects of public
awareness of SFIP?
./ Customer Satisfaction:
Do users find SFIP's data useful, understandable, and easy to access?
./Behavior Change -- Information Use:
How are SFIP data being used?
* Goal 2: Improve multimedia facility profiling and sector-based analysis
./ Advancing Data Integration Knowledge:
What has SFIP shown about the feasibility and benefits of creating facility and
sector-level environmental profiles?
./ Assessing Data Quality in Federal Data Systems:
What have SFIP's ongoing data quality assurance procedures shown about EP A
data quality?
Future Directions: Comments and Recommendations
10
December 1999
-------
Sources of Data
EP A has continued to seek input from SFIP stakeholders throughout the development and
implementation of the project. This evaluation draws on comments received during the project's
planning process and since the website's launch, as well as findings from numerous other
sources, including a series of discussion groups conducted specifically for this evaluation during
1999. The major sources are described below:
Hotline calls and em ailed comments
Since the website's launch, SFIP has maintained a user support hotline staffed during business
hours. The project maintains a record of each call. SFIP's website also allows users to submit
comments by email at any time. Users submitting comments via the SFIP Hotline and/or by
email are generally not required to identify themselves.
Website use statistics
Information about SFIP users which can be tracked on the website includes statistics on the
frequency of "hits" I and "user sessions"210gged on the site (see Appendix 1 for detailed reports
on SFIP use statistics for May I-October 3, 1998).
Regional/State Demonstrations and Discussion
EP A headquarters staff have made presentations on SFIP to Regional and State officials at all ten
Regional EP A offices since the SFIP website was launched. These presentations had multiple
objectives: to explain the purpose and goals ofSFIP; to conduct an online demonstration of the
project's website; and to obtain feedback on how the SFIP can be developed further, including
what additional information and additional sectors would be most valuable to the Regions and
states. Managers and staff with responsibility for data management, environmental
measurement, enforcement, compliance assistance, environmental justice, and government
relations attended the presentations.
In order to most effectively demonstrate how Regional and State staff might use SFIP, each
presentation included a review of the environmental history and performance of a facility within
the Region. Attendees were shown a sample of the vast array of information available on the site
and saw how SFIP allows users to choose from among several different ways to sort and analyze
the data presented. During the majority of the demonstrations, Headquarters staff also provided
analyses ereated from SFIP downloadable spreadsheets that allow the Regions to compare their
programmatic efforts and their facilities' performance to national averages.
Discussion groups
EP A organized discussion groups with (l) EP A staff, (2) representatives from environmental
organizations, (3) representatives of facilities included in SFIP, and (4) representatives of state
agencies. These meetings provided an opportunity to obtain input from frequent users of SFIP
IA "hit" is an action performed on the website, such as viewing a page or downloading of a file.
2 A "user session" is a discrete period of activity generated by an individual user.
11
December 1999
-------
and to explore their experiences with and reactions to the project. The discussion groupS were
convened in Washington, DC during 1999. SFIP project staff were not present at any of the
discussion groups, and participants were assured that their remarks would remain anonymous. In
keeping with the Paperwork Reduction Act, all meetings included fewer than ten people.
Subsequently, several individuals who were unable to attend the group meetings provided input
through separate interviews. In this report, the term "discussion groups" will be used to refer to
both the group meetings and the individual interviews. For sample questions which discussion
group participants were asked to consider, see Appendix 2.
In addition, as part of an ongoing dialogue with the trade associations representing facilities
included within SFIP, EPA Headquarters staff met with the associations on March 11, 1999 to
obtain their current perspective on SFIP and to gather insight from their experience in working
with member facilities on the project. The meeting was attended by representatives from the
Association ofIntemational Automobile Manufacturers (AIAM), The American Forest & Paper
Association (AF&PA), the American Petroleum Institute (API), the National Petroleum Refmers
Association (NPRA), and the Aluminum Association (AA).
12
December 1999
-------
EVALUATION RESULTS
Goal 1:
Provide greater public access to compliance and
facility-level information
Three main topics are considered in evaluating SFIP's public access goal:
Awareness
SFIP's success as a community right-to-know initiative depends on public awareness of its
information and tools. This section evaluates the extent of public awareness of SFIP
materials, and also briefly discusses the potential effects of this awareness.
Customer Satisfaction
This section evaluates whether the SFIP website provides the public with an effective
information tool by asking the following questions:
Are the data provided by SFIP valuable to users?
Is the information presented on the SFIP website understandable by users?
Is the website easy to find and use (i.e., "user-friendly")?
Information Use
This section discusses how SFIP's information is being used, both by members of the public
and by EP A.
13
December 1999
-------
Awareness
EP A has promoted awareness of SFIP primarily through its work with project stakeholders in
designing the project and ensuring data accuracy. In the course of developing the environmental
profiles presented on the website, EP A worked closely with state governments, Regional EP A
personnel, environmental organizations, facilities from the five industry sectors included in SFIP,
and trade associations for those industry sectors. A press release announced the May 1, 1998
launch ofthe site by EP A Administrator Carol Browner (Appendix 3). Over 50 articles on SFIP
have appeared in newspapers, magazines, and trade journals. Many websites offer links to SFIP.
In addition, Federal Register notices have notified the public about the project's major
milestones, including a May 1997 public meeting at which EP A solicited stakeholder input.
Finally, EPA headquarters staff have presented the project and provided numerous
demonstrations of the website to trade associations, environmental organizations, and state and
Regional staff.
FINDINGS:
Over 66,000 user sessions and 404,000 hits have been logged on SFIP's website since May 1,
1998.
Hotline calls and emails indicate that there are numerous users of many types, including
individual citizens, and representatives from environmental groups, law firms, and consulting
firms.
SFIP is an innovative tool new to the Agency; thus, continued outreach efforts should be
emphasized.
Extent of Awareness
Since the SFIP website's launch, website use statistics show that users have logged a total of
66,924 sessions on the site, or an average of 4, 183 sessions per month. The site received 404,642
hits during this period. In the site's first month, there were 14,375 user sessions and 95,379 hits.
Subsequent use has been steady, with an average of3,503 user sessions and 20,618 hits per
month (Graph 1). A September 1998 summary of use statistics for all Office of Enforcement and
Compliance Assurance (OECA) web pages shows that SFIP is among the projects which have
drawn the greatest number of users to OECA sites.
An increase in website use as measured in both user sessions and hits has been consistently
observed during each week following an update of SFIP' s data. In response to this clear
indication that users are interested in accessing new data as soon as it becomes available SFIP
. ,
recently added a page to the websIte that allows users to request an automatic email notification
when a data refresh has occurred.
14
December 1999
-------
Graph 1: SFIP Web Use Trends
User Sessions (Left Axis) and Total Hits (Right Axis)
16,000 120,000
14,000 --; c::J User Sessions
-+- Hits 100,000
I
12,000 -I
I 80,000
en 10,000 -,
c:
0
en
en en
Q) 8,000 -, 60,000 :!::
CI) J:
..
Q)
en
:J 6,000
40,000
4,000 ~-
2,000 20,000
0 -; 0
9;)'0 9;)'0 9;)'0 9.J'O 9;)'0 9;)'0 9.J'O 9.J'O 0)0) 9.J0) 0)0) 9.J0) 9.J0) 9.J0) 0)0) 9.J0)
~ts\ ':,v<:! ':,~ ,,?-vC$ ~0
-------
SFIP's launch was widely reported in mainstream newspapers, and in industry and government
print and Internet publications. Twenty-three articles which appeared in May 1998 were mainly
news reports that described the project and announced the availability of the website. A number
of analyses which express different perspectives on the project have also been published. The
Working Group on Community Right-To-Know issued a press release describing SFIP as "an
important advance for public access to EP A information." The Environmental Defense Fund
also issued a press release which praised the project. In contrast, articles in publications such as
Octane Week and Business Insurance have emphasized industry's concern with SFIP's data
quality. Over 30 additional articles on SFIP have appeared since the May 1998 launch of the
website.
EP A headquarters and Regional staff and environmental organization representatives who
participated in the 1999 discussion groups reported that they have begun using SFIP for various
analyses and are becoming increasingly knowledgeable of the array of information that can be
accessed through the website. Environmental groups also reported that projects for which they
use SFIP data often involve partnerships with other organizations and communities, whose
members may become aware of and use SFIP in the course of the collaboration. The "public"
they work with who are affected by SFIP includes local-level activists and community groups,
policy makers, and people who live near particular facilities.
Organizations that have used their websites to link to SFIP and to provide information on the
project also help to increase public awareness of SFIP. For example, the Environmental Defense
Fund (EDF) not only provides a link to SFIP from their "Scorecard" site, but also gives an in-
depth explanation ofthe project and states that "EDF encourages members of the public to use
this new, innovative right-to-know tool." The State of Michigan, which initially expressed
concern over SFIP, now lists SFIP on its website as an "Environmental Link ofInterest."
Awareness of SFIP has also increased due to outreach efforts by EP A headquarters staff. A
variety of organizations have expressed interest in SFIP and have requested presentations on the
project. In addition to speaking with representatives from facilities, environmental organizations,
and state agencies that had been active in the development phase of SFIP, EP A also met with
such organizations as the National Electrical Manufacturers Association, the Lead Industries
Association, and the Forum on State and Tribal Toxics Action (FOSTTA), as well as
Congressional and State staff. EP A headquarters staff also visited all EP A Regional offices to
introduce the Regional staff to this new tool.
In discussion groups, industry representatives reported that, although they had used SFIP to
check the accuracy of EP A data, they had not heard from the public regarding SFIP. Similarly, at
the March 1999 trade association meeting, representatives understood their members to be
keeping abreast of their individual facility information, but were unaware of communications
between the facilities and the general public with regard to SFIP.
At least one company with several facilities included within SFIP has used the project's data to
communicate with the public via the company's public relations activities and its own website.
16
December 1999
-------
ASARCO Incorporated has released separate press releases for five facilities discussing the data
that appeared for each facility within SFIP3
During SFIP's development, some states expressed concern during SFIP's development that they
would be burdened by numerous inquiries from the public and facilities about SFIP data.
However, state representatives who attended the 1999 discussion group reported that this has
not, in fact, been the case. Instead, participants commented that the lack of contact between their
agencies and the public concerning SFIP indicates that the public may not be sufficiently aware of
SFIP to derive its full benefit. Discussion group participants noted that making environmental
data accessible to the public is a significant priority for their states and all described initiatives
underway at their agencies to increase the public's access to data, particularly through the
Internet.
Effects of Awareness
Input from all types of SFIP stakeholders suggests that increased public access to facility
environmental performance data in general, and greater public awareness of SFIP in particular
may provide an additional incentive for companies to maintain exemplary environmental records
and may increase the dialogue between included facilities and the surrounding community.
EP A staff who participated in the 1999 discussion group, as well as Regional staff who attended
website demonstrations, felt that improved public information disclosure can and does bring about
improved environmental performance, stating that "heightened public scrutiny can heighten
industry stewardship." State representatives who participated in the 1999 discussion groups also
felt that SFIP may prompt companies to increase their compliance efforts. In addition, EP A staff
have reported that SFIP has promoted greater state/Regional cooperation. EP A staff felt that
some state agencies have become more responsive in making data corrections. Regional staff
who attended demonstrations of SFIP also expressed a need to develop innovative approaches
and to go beyond "traditional enfQrcement" in order to bring industry into compliance. The
Regions are looking to tools such as SFIP to achieve improved environmental performance when
the potential for environmental gains through additional enforcement efforts is limited.
Representatives of environmental organizations who participated in the 1999 discussion group felt
that analytical reports they produce using resources such as SFIP generate increased attention
from local press and calls to facilities with less than perfect records. Several environmental
organization representatives discussed mechanisms by which public access to environmental
information could result in pressure on companies to improve their performance, such as public
meetings.
Industry representatives have described a heightened awareness of public perception as the major
change in facility behavior due to SFIP Discussion group members identified relationships with
3Press releases dated May 1,1998, www.asarco.comlPressReleases/sfipamar.pdf.
www.asarco.comlPressReleases/sfipehel. pdf, www.asarco.comlPressReleases/sfipelps.html,
www.asarco.comlPressReleases/sfipglov. pdf, Www.asarco.comlPressReleases/sfiphay. pdf,
referenced October 15, 1999.
17
December 1999
-------
communities sUlTounding their facilities, including with facility employees and their families, as
a high priority for their companies. They stated that their companies are particularly concerned
with how they are perceived by these members of the "public," as well as by their customers and
wish to maintain a positive image. During the March 1999 trade association meeting, an
association representative for the pulp manufacturing sector reported that SFIP can be seen as a
positive innovation in so far as it is viewed by some facilities as providing an opportunity to
interact more with the surrounding community.
18
December 1999
-------
Customer Satisfaction
SFIP's website is designed to provide public access to useful, understandable environmental
information in a user-friendly format. This section evaluates three components separately,
asking:
Are SFIP data useful?
Are SFIP data understandable?
Is the website easy to find and use?
Although a hard copy report was made available in September 1998, the Internet is the primary
pathway for public access to SFIP data. Three major types of reports are available, which present
"raw" data for each facility and aggregated data by facility and by sector. Background
information including detailed descriptions of the data indicators and a summary of the project's
history provides context to enhance users' understanding ofthe data presented. "Easy Search"
pages allow users to locate specific facilities and groups of facilities by city, state, and EP A
Region, as well as by facility name, while the powerful "Custom Search" option allows users to
identify and sort facilities by different data indicators, such as whether the facilities have reported
pollutant spills, or the number of people who live within three miles of the facility. Spreadsheets
containing the raw data accessed through these searches are downloadable in two formats (Lotus
123 and Microsoft Excel).
EP A has also continuously added new features and incorporated suggested changes to the SFIP
website during the project's development. For example, the ERNS Incident Report available
through SFIP has been reformatted to make it easier for users to quickly identify key data
elements such as the type and quantity of material released during pollutant spills. SFIP Detailed
Facility Reports have also been reformatted to increase ease of use, e.g., by presenting
compliance history by calendar quarters rather than by EP A's fiscal year quarters. Similarly,
explicit date ranges for the two-year data presented in the Facility-level Statistics report have
been added to the headings of that report. New website features include a page describing SFIP's
update schedule and procedures, a page with background information on SFIP's "facility
universe", and a form that allows users to request email notification when SFIP data are updated.
FINDINGS:
Users value the data presented in SFIP - the public is logging on to SFIP and "digging down"
to the wealth of data offered.
Most users feel the site is user-friendly. However, some users have called for continued
improvement in providing "context" for SFIP's data and better explanations for some data
elements.
Requests from users for access to data for more industry sectors in addition to the five
currently included in SFIP are a strong indication of customer satisfaction with the pilot
project.
19
December 1999
-------
Are SFIP data useful?
EPA tracks website use statistics for SFIP to assist in evaluating the project and generating
measures that show how SFIP's data are used by the public. Knowing the number of "hits" to the
website is helpful but EPA understands the need to go beyond "counting beans" to measure
environmental results. Thus, SFIP developed additional measures to assess users' experience with
the database and understand how the database is being utilized. Use statistics tracked for the
SFIP website show that the majority of hits to the site relate directly to data use (e.g., hits to
search pages and forms, data downloads, and data report views), while a smaller proportion of
users' activity is devoted to accessing information about the project's purpose and background
Graph 2 shows estimated data use since the site's launch.
Graph 2: Proportion of Hits Related to SFIP Data Use
£
X.
"re
4->
o
o
0)
G)
|
0)
e
0>
OL
100%
90%
80%
m Estimated use of other data pages, including search and explanation pages
n Estimated Use of Detailed Facility Reports
$>
& <§> $> <$> <£>
<$>
Month
It is estimated that data use accounted for approximately 72% of all hits over this period, and has
increased since the site's launch from 64% in May 1998 to 82% in July 1999 and 76% in August
1999 In addition, while the estimated proportion of views of Detailed Facility Reports for
individual facilities has increased, other forms of data use have declined, probably reflecting more
efficient use or bookmarking of search paths by repeat users Once users become familiar with
the SFIP website, they have sufficient knowledge to go directly to the data rather than repeatedly
reviewing the background and explanatory information.
The website's use statistics also indicate users' level of interest in each industry sector included in
SFIP As measured by hits to each sector's dedicated "Data Access" page, the Petroleum
20
December 1999
-------
Refining sector data has received the most attention to date, followed by Pulp Manufacturing,
Iron and Steel Mills, Primary Nonferrous Metals Smelting and Refining, and Automobile
Assembly (Table 1). However, Primary Nonferrous Metals and Automobile Assembly facilities
received the highest number of hits per facility.
a e : aa ccess IY ec or
Number of
Cumulative Facilities in
Page Viewed Hits Sector Hits/Facility
Petroleum Refining Data Access 7,058 168 42
Pulp Manufacturers Data Access 5,092 244 21
Iron and Steel Mills Data Access 3,944 121 33
Primary Nonferrous Metals Data Access 3,171 50 63
Automobile Assemblv Data Access 2,912 57 51
T bl 1 D t A
b S t
The SFIP website is considered useful by staff from EP A headquarters, Regional EP A offices,
and state agencies. A September 1998 report by the General Accounting Office which assessed
the status ofEPA's efforts to provide communities with risk information, make TRI data publicly
accessible, and develop standards and procedure for disseminating environmental information
found that SFIP is one of three EPA projects that "collectively... would substantially expand the
information available to communities"4.
Both EP A headquarters staff who participated in the 1999 discussion groups and Regional staff
who attended demonstrations of SFIP felt that reliable, externally referenced universes are an
extremely valuable feature of SFIP. SFIP data has allowed these users to screen and target
facilities for compliance assistance and enforcement, access background information on facilities
of particular interest, and identify trends in sector data. Following presentations on SFIP
capabilities and demonstrations of the website's capabilities, Regional staff saw SFIP as useful
for targeting and compliance assistance efforts. In addition, Regional staff felt that SFIP could be
a catalyst for increased dialogue and cooperation with states in efforts to ensure the accuracy of
the underlying databases.
State representatives who participated in the 1999 discussion groups were not aware of staff use
of SFIP within their agencies. However, states provided positive feedback at SFIP
demonstrations. For example, a representative from the Oregon Department of Environmental
Quality considered SFIP to be the type of project EP A should continue, and pointed out that SFIP
is beneficial to states as a planning tool. A Maryland inspector who conducts multi-media
inspections said that SFIP can be useful in deciding what facilities to inspect. He commented
that it is "very useful to first go on the computer and review information before going out to a
facility." A Florida state representative commented that SFIP "is excellent" and the Agency's
efforts on this project should be "applauded."
4United States General Accounting Office, Environmental Information - Agencywide Policies
and Procedures are Neededfor EPA 's Information Dissemination, GAOIRCED-98-245,
September 24, 1998.
21
December 1999
-------
Environmental organizations have stated that they particularly value the complete, "quality-
checked" universe of facilities provided by SFIP. In discussion groups held in 1999, SFIP's
well-defined sector universes and facility permit linkages were described as "incredibly useful,"
in contrast to other sources where it may be "impossible to determine what constitutes a sector,"
or where differing permit names and addresses make it difficult to identify all the permits
belonging to a given facility. Environmental organizations stated that while facilities differ, SFIP
provides a valuable starting point for comparisons.
Other aspects of SFIP that environmental representatives have found particularly useful include
the following:
. SFIP provides demographics and detailed facility reports, which may be useful to
community-level groups that want to know what's in their "backyards"
. Compliance information
. SFIP is unique in providing facility production data
Environmental organization representatives have expressed a strong desire to have access to both
additional "raw" and analyzed data. One representative commented that SFIP data is "not raw
enough." For example, while SFIP provides an indicator that shows if a facility is considered to
be in Significant Noncompliance (SNC) for any program, environmental organizations would
like access to the underlying data used to determine SNC status. Additional examples of "raw"
data these representatives would like to access include: expiration dates for permits; permit
limits; more detailed information on violations, such as whether they have been contested by the
facility; and information on the CAA attainment status of the facility's location. Environmental
representatives stated that data analyses are also valuable, as "sometimes raw is too much to get
into." They would like EPA to share any and all analyses performed by the Agency.
Industry representatives who participated in the 1999 discussion group indicated that direct
access to EP A's main databases via SFIP could be useful to them. These industry representatives
were generally supportive ofSFIP's public access goal, stating: "It's good to pull together the
programs in one place." These representatives felt that SFIP is a positive step forward and could
be an even more "valuable tool if made more user-friendly in terms of understanding the data
and if it were easier to get corrections made."
Industry feedback indicated that some companies have tried to develop indicators which allow
comparison of environmental performance within or across facilities or companies, suggesting
that the summary statistics provided by SFIP might be potentially useful to them. Industry
representatives also praised SFIP Facility-level statistics which present CW A permit exceedences
as a proportion of reports as an example of a useful and objective measure. Industry
representatives stated that their use ofSFIP is limited due to concerns with data quality. In
addition, they noted that they keep their own compliance records and do not see any additional
benefit from being able to obtain data from the SFIP website. However, trade association
representatives who met with EP A in March 1999 stated that data in the underlying databases
which SFIP integrates has improved in quality as a result of the increased accessibility to this
information provided by SFIP, and suggested that increased reliability of the data makes it more
22
December 1999
-------
useful. Participants felt that the data's accessibility may provide the means for some facilities to
increase their communication with the public. Additionally, a trade association representative
reported that some petroleum refineries have indicated that they value information on CW A
exceedence and like how this information is presented in SFIP.
An additional indication that users value the kind of data presented by SFIP is that many users,
particularly staff from EP A's Regional offices, have expressed strong interest in being able to
access data for other sectors through the SFIP website, including metal services, electric utilities,
paper manufacturers, and federal facilities. In addition, Hotline users have submitted suggestions
for additional sectors, including hazardous materials treatment facilities and large scale animal
feeding operations. These and other suggestions for SFIP's expansion are discussed in more
depth in the last section of this report.
Are SFIP data understandable?
Of 295 Hotline calls and emails received since May 1, 1998, approximately 50 have been
questions about how to interpret SFIP's data. For the most part these questions have been
quickly and easily addressed by referring users to pages on the site which describe the SFIP
indicators and data reports. Facilities requesting additional information about their own data
reports have been referred to Regional contacts. In a few cases, Hotline staff have performed
additional research in order to help users interpret quarterly compliance codes for Clean Water
Act permits and Clean Air Act permits. Several users have requested clarification on whether
particular indicators represent averages or cumulative figures. In addition, two users had
questions about how to understand the demographic statistics presented in the Detailed Facility
Report.
Both EPA headquarters and Regional staff have stated that SFIP's data are structured in a way
that should be understandable to the public. They also felt that SFIP provides valuable
background information that enhances the public's' understanding of what is occurring at a given
facility. The SFIP production data was cited by many participants as an example of valuable
context provided by the site because it makes facility comparisons more meaningful by providing
a rough measure of facility size and output. For example, users may compare each facility's ratio
of chemical releases and transfers per unit produced as well as comparing the absolute amount of
chemicals released and transferred. However, Region 2 wants actual- would be even more
useful.
During SFIP's development, states expressed concern that the public might misinterpret SFIP
data. They have continued to emphasize the importance of providing explanatory context for
data. However, 1999 discussion group participants noted that the public has not contacted their
agencies with questions about SFIP and they reported no instances of data misuse. One
participant stated that "the fact that we haven't had any calls shows that concern that the public
will panic was unwarranted". While states expressed appreciation for the extensive
documentation provided to help users understand data on the SFIP website, some participants felt
that the website might be overly complex for the general public and some were doubtful that data
"caveats" will be read and used.
23
December 1999
-------
Environmental organization and state agency representatives who participated in the 1999
discussion groups also emphasized the importance of explanatory information that allows users
to understand data in context. While some praised SFIP for providing "really good" and "very
useful" context, others felt that some SFIP data are difficult to understand and/or not useful.
The SNC designation, in particular, is considered problematic. One discussion group member
commented that SNC is not valuable to the public because it is not an intrinsically meaningful
statistic, but rather, "an EP A-created term that doesn't mean anything to anyone." In order to use
it, this person felt, a user has to "dig in, get the guidances, talk to people to find out what they
think it means." Similarly, one participant suggested that the Detailed Facility Report should
include a placeholder for facilities with no enforcement actions so that it is clear to users that the
section has not been inadvertently omitted.5
Initial industry concerns regarding contact from members of the public who may have
misinterpreted the data have not been substantiated. Facility and trade association
representatives who participated in the 1999 discussion groups did not cite instances in which
facilities have expended resources to dispel public misinterpretation of the data.
However, while industry representatives felt that the context and interpretation provided by the
SFIP website makes it "a database with an attitude, " they also stressed that some SFIP data
could be misleading to the public without additional context to provide a complete picture of
environmental performance at a given facility. They suggested that additional information on the
type and duration of violations would help SFIP's users identifY those compliance problems that
involve a greater risk to the public. Similarly, industry commented that aggregated or summary
statistics are not always comparable when facilities have very different processes.
Industry also stated that CW A Historical compliance data presented in the Detailed Facility
reports were difficult to understand. Representatives felt that SFIP's Data Dictionary offered
inadequate explanation of quarterly permit level compliance codes and specialized terms such as
"Discharge Monitoring Report" (DMR). One commenter questioned the public's ability to use
SFIP, "if industry can't understand their own data." For example, it was felt that '"the public
doesn't know what a DMR is." In addition, industry representatives have suggested that SFIP
provide links from the different sections of the Detailed Facility Report directly to the
corresponding explanations in the Data Dictionary6.
5 As a result of such feedback, SFIP's Detailed Facility Report is currently being revised. A
revi~ed version which .wi!l inclu.de section placeholders where needed and links to specific
sectIOns of the Data DIctIonary IS expected to be available to the public in an upcoming data
refresh.
6See above.
24
December 1999
-------
Are SFIP data easy to find and use?
Based on comments received through the SFIP Hotline and by email, users have found the SFIP
website and its features accessible and user-friendly Only ten of the over 200 users who have
called the SFIP Hotline or sent emails from the site required assistance in using SFIP's user-
defined searches In eight of these ten cases, users were quickly able to orient themselves after
Hotline staff walked them through a sample search In the remaining two cases, users did not
reply to emails from Hotline staff asking for further information about the nature of their
difficulty, suggesting that the users may have resolved their questions in the interim
EP A headquarters staff who participated in the 1999 discussion groups generally felt that the site
is easy to navigate However, they suggested that the "Disclaimer and Limitations" section of the
Data Access pages should appear only when users are ready to submit a query Regional and
state staff who attended website demonstrations at the Regional offices found the website to be
"user-friendly" and capable of assisting in Regional analysis It was commented that SFIP "brings
information to the surface" in an understandable format which provides a starting point for
discussion By making information accessible in an easy-to-use platform, SFIP enables all
stakeholders to see where potential compliance problems exist and thus can potentially facilitate
greater agreement on priorities for enforcement and compliance efforts
Environmental organizations stated that having electronic access to different kinds of
environmental data in one place is valuable because such access saves time and effort, making
analysis easier Because of the convenience provided by SFIP's downloadable spreadsheets, one
participating organization prefers to use TRI data from SFIP, although it is available from other
sources Participants also valued the ability to search multiple ways for a particular facility and to
sort data for numerous facilities by more than one indicator
All trade association meeting participants reported that the SFIP website is thought to be user
friendly and easy to search for information All were satisfied with the website as it currently
appears and did not believe any changes were necessary Industry representatives who attended
the 1999 discussion group did not have any negative comments on the ease of navigating the
website However, they did suggest that SFIP would be easier to find if the EP A's Home Page
provided a direct link to SFIP In addition, some industry representatives commented that the
Detailed Facility Report is difficult to read7
7 A revised version of the Detailed Facility Report which will be available in an upcoming refresh
has been resized and reformatted to improve readability and navigability
25
December 1999
-------
Information Use
SFIP data is valued and thus used for many reasons. SFIP makes EP A data uniquely accessible
through a single Internet database and provides extensive documentation to enable users to
understand the information presented. In addition, SFIP offers indicators that allow users to
better "digest" the complex layers of data taken from the underlying databases. Finally, SFIP's
data is the product of a thorough quality-assurance process.
The potential results of such use were discussed by participants at recent East and West Coast
Conferences co-sponsored by EP A and the National Partnership for Reinventing Government.
The conferences were attended by representatives from environmental and community groups,
trade associations, small and large business representatives, academics, and federal, state, local,
and tribal governments. Attendees felt that public access to facility environmental performance
data is one factor which can influence facilities to improve their compliance record. Conference
attendees also discussed the need for OECA to move beyond enforcement by developing
innovative tools such as SFIP which also have compliance assistance applications. It was
commented that "OECA has historically focused on enforcement, but the mission of OECA
should focus not just on enforcement but also on... meaningful compliance assistance programs."
In addition, internal Agency use of the data means that SFIP functions as a sort of laboratory for
data integration initiatives.
This section augments findings from the two preceding sections (Awareness and Customer
Satisfaction) by presenting more detailed information about specific uses of SFIP data.
FINDINGS:
EP A and environmental organizations have undertaken numerous analyses using SFIP data.
SFIP facilities primarily access SFIP to verify the data in their own profiles, but at least one
company, ASARCO Incorporated, has published several analyses based on the SFIP data.
26
December 1999
-------
Use by Environmental Organizations
Environmental organizations have used SFIP to obtain reliable universes of facilities in several
sectors; access Clean Air Act violations across all sectors; and assess both facility and regulatory
performance. Examples follow.
Council on Economic Priorities (CEP)
The Council on Economic Priorities (CEP) is a non-profit organization that has developed
reports on more than 320 corporations, providing "information about corporate social
responsibility for consumers and investors." CEP also publishes reports on particular issues and
industry sectors (e.g., the petroleum industry, the automobile assembly industry), as well as a
consumer guide called Shopping for a Better World.
Most recently, CEP used SFIP in developing a report entitled "The Worst and Best Auto & Tire
Companies." This report, which analyzed eight automobile assembly companies and five tire
manufacturers, was sent to all CEP members. To research this report, CEP used SFIP, the Toxics
Release Inventory (TRI), and the Aerometric Information Retrieval System (AIRS). CEP
downloaded information from SFIP about pollutants for each facility in the universe of auto
assemblers and aggregated them by company to develop a ranking of the most-polluting and
least-polluting companies.
Ecology Center of Ann Arbor
The Ecology Center of Ann Arbor is a grassroots environmental organization that works to
promote clean air, safe water, healthy communities, and environmental justice. The Center has
used SFIP to gather compliance data for the automobile assembly sector and often tracks the
performance of individual facilities.
Environmental Defense Fund (EDF)
The Environmental Defense Fund (EDF) is a national non-profit organization with 300,000
members. EDF used the newly launched SFIP website to analyze the incidence of significant
Clean Air Act violations and of pollutant spills for each of the industry sectors in the pilot. In
May 1998, EDF published a detailed three-page news release entitled "Environmental Defense
Fund Analysis of Sector Facility Indexing Data Finds Violators." The news release ranked the
sectors by number of violations and spills. According to one EDF staff member, SFIP "provides
the public with an excellent means to identify, for example, facilities with the greatest population
density and environmental justice concerns, facilities with the best and worst enforcement, and
spill records federally or within a particular state, and the states performing the most
inspections." Another staff member stated that SFIP "is a model of government responsiveness to
the public's need for accessible, facility-specific, environmental protection information."
On another project using SFIP data, EDF worked with Friends of the Earth (see below) to
analyze the records of facilities in the iron and steel industry. EDF also used SFIP to help
"normalize" data in the five pilot project industry sectors for EDF's own database of industrial
facilities, "Scorecard."
27
December 1999
-------
Environmental Working Group
The Environmental Working Group (EWG) is a non-profit organization which provides
information and policy analysis to the general public, environmental organizations and other
public interest groups, journalists, and policy makers EWG recently used the facility universes
defined by SFIP to analyze enforcement efforts and facility compliance with the Clean Air Act for
the five sectors included in SFIP EWG published "Above the Law" in May 1999, and "Above
the Law How California's Major Air Polluters Get Away With It" in July 1999
Friends of the Earth (FOE)
FOE is a national non-profit organization that works on environmental and citizen empowerment
issues FOE focuses on (1) economics, (2) community, health, and environment, and (3)
international issues
For one project - the Clean Steel Coalition - FOE used SFIP data to help analyze the records
of iron and steel facilities The information was given to community activists working with the
Coalition, to help them understand the technical issues and monitor industry activities in their
communities Also, with the aid of such information, citizen activists attending permitting hearings
for iron and steel facilities were better equipped to understand the technical aspects of the
hearings FOE has also
. Obtained TRI data ITom SFIP for the universe of iron and steel facilities and ranked
production efficiency (e g , how much zinc released/ton of steel)
. Reviewed the demographic information provided in SFIP to study the distribution of
minority populations in communities surrounding SFIP facilities
Use by EPA
EP A staff have made use of SFIP for multiple types of analyses, including screening and targeting
facilities for compliance assistance and enforcement, accessing background information on
specific facilities, and identifying trends in sector data Examples of analyses undertaken by EP A
staff using SFIP data include the following
. Identification of TRI non-reporters, facilities under -reporting or failing to report TRI
chemicals commonly reported within their sector, and facilities with unusually high or low
frequencies of reported pollutant spills
. The Office of Compliance and Office of Regulatory Enforcement have used SFIP data as a
baseline for trend analysis for the Iron and Steel, Nonferrous Metals, and Petroleum
Refining sectors and for developing Sector Strategies for these industries
. Analyses prepared from SFIP's downloadable spreadsheets have been used by EP A
Regions to compare their facilities' performance to national averages
28
December 1999
-------
. Analyses prepared from SFIP's downloadable spreadsheets have been used by EPA
Regions to compare their facilities' performance to national averages
Use by Industry
In the 1999 discussion groups, industry representatives stated that their primary use of the site is
to review their own facility data. They also described having accessed SFIP data reports for
competitors' facilities and having analyzed data for their industry sector to assess which facilities
appear to be "bad actors." Some representatives described their own companies' efforts to
develop internal environmental measures and to provide environmental information directly to the
public (e.g., through their own web sites ), and said that indicators of environmental performance
are potentially valuable. But they stated that due to doubts about SFIP's data quality, they would
use internally developed data or obtain data from trade associations rather than use SFIP for
benchmarking.
However, at least one company has demonstrated that it believes SFIP data are of sufficient
quality for use in its own analyses. As previously mentioned, ASARCO has studied SFIP data for
its copper processing facilities and has published several analyses to provide the public with
additional information "to enhance understanding of SFIP data8" ASARCO feels that SFIP data
show that its employees "are doing an excellent job of protecting and improving the
environment,,9
8ASARCO press release dated May 1, 1999, www.asarco.comlPressReleases/sfipglov.pdf.
referenced October 15, 1999.
9 ASARCO press release dated May 1, 1999, www.asarco.comlPressReleases/sfipamar.pdf.
www.asarco.comlPressReleases/sfipehel. pdf, www.asarco.comlPressReleases/sfipelps.html,
www.asarco.comlPressReleases/sfipglov. pdf, www.asarco.comlPressReleases/sfiphay. pdf,
referenced October 15, 1999.
29
December 1999
-------
Goal 2: Improve multimedia facility profiling and sector-
based analysis
The evaluation ofSFIP's efforts to define industry sectors, develop facility universes for those
sectors, and create whole-facility environmental profiles for each included facility includes a
brief summary of the data integration activities undertaken. Feedback from facilities and state
and Regional staff during the Data Quality Assurance Review performed between November
1996 and October 1997, shows that pennit information can be successfully compiled and
summarized on the facility level. SFIP has also been successful by showing that an accurate
accounting of the universe of facilities belonging to a given sector is feasible and enhances the
value of sector-based analyses, which are increasingly being used by EP A and others to improve
environmental management and compliance of the regulated community. In addition to
documenting the demonstrated feasibility of such data integration procedures, this section also
summarizes the comments of discussion group participants interested in improving
environmental data integration.
Advancing Data Integration Knowledge
This section examines how SFIP has advanced EP A's data integration knowledge, by asking
the following question:
- What has SFIP shown about the feasibility and benefits of developing reliable facility
universes for defined industry sectors and assembling complete permit linkages for a given
facility?
Assessing Data Quality in Federal Data Systems
This section discusses the quality of the data compiled by SFIP. Findings are divided into
the following sub-sections:
- SFIP's pre-launch Data Quality Assurance Review
- Quality Assurance Procedures for Updating SFIP Data
30
December 1999
-------
Advancing Data Integration Knowledge
Data integration efforts within SFIP were designed to achieve multiple objectives: to develop and
demonstrate procedures for defining industry sectors and determining the universe of facilities in
a given sector; to establish procedures for building complete facility records; and to assess the
value of such facility- and sector-level information.
The Problem
Defining which facilities belong within a sector: Several factors make it difficult to reliably
identify the universe of facilities belonging to a given industry sector using EPA's data systems.
For example, EPA does not collect process-specific information as part of permitting activities
and SIC code information in EPA's systems is inconsistent (some data systems have up to six
SIC code fields and there is little guidance on entering SIC codes).
Establishing which permits belong to afacility: At the start ofSFIP, EPA's Facility Indexing
System (FINDS) was the established means of linking the permits and other identifiers of
facilities together. While FINDS depended primarily on physical addresses that were in common
with the RCRA permit address, it had been inconsistently maintained over the years.
Consequently, FINDS-based facility records were incomplete. For example, as many as 50% of
PCS permits were not linked to FINDS IDs and facility permits were frequently split among
more than one FINDS ID, sometimes resulting in as much as a fourfold overcount of facilities in
a given sector. Thus, at the facility-level, FINDS resulted in variable amounts of inappropriate
inclusion and exclusion of permits.
The SFIP Approach
To develop reliable facility universes for each sector, SFIP first established definitions for each
sector based on similar, specific manufacturing processes. For example, SFIP defines the "Pulp
Manufacturing" sector to include facilities producing pulp from virgin materials and facilities
that pulp and de-ink secondary (recycled) fiber. SFIP then defined a "facility" as any physical
location with one of the defined processes in operation, even if it is not the primary operation.
To develop a list of facilities which belong in each sector, SFIP identified independent sources of
information about each industry which provided information about the processes used at each
facility, facility name, and location.
To establish a set of complete and accurate permit linkages for each facility, SFIP verified and
augmented the linkages available through the FINDS system through manual searches ofEPA's
program databases. In addition, Regional and state staff as well as the facilities themselves
reviewed the data reports for completeness and accuracy.
31
December 1999
-------
FINDINGS:
SFIP successfully developed and implemented protocols for developing a reliable facility
universe for a given industry sector.
SFIP successfully developed and implemented protocols for establishing complete and
accurate permit linkages for individual facilities to enable the creation of whole-facility
environmental profiles.
Sector Universes
SFIP has assembled sector universes which are more accurate than those available directly from
the individual underlying data systems and those previously available through the Facility
Indexing System (FINDS). Selecting facilities based on SIC code associated with the permits
linked by FINDS can result in as much as a fourfold over count of facilities. During the SFIP
Data Quality Assurance Review, 18 of the 661 candidate facilities claimed to be out of scope.
Following review of documentation, SFIP was launched with 651 facilities -- 98% of the
facilities initially identified. Since the launch, the universe of facilities has been updated by: (1)
including newly operating facilities and some facilities that self-identified or were identified by
EP A staff or a third party; and (2) excluding facilities that closed in 1995 or 1996 and a number
of secondary copper refineries. In total, 19 new or additional facilities have been added to the
database, and 30 have been removed.
Permit Linkages
SFIP's permit linkages are accurate and improve on the FINDS linkages. SFIP has reduced the
number of erroneously linked permits and captured virtually all of these facilities' permits that
had not been linked via FINDS. Permits of several co-located operations (e.g., TRI submissions
for on-site energy production, sawmills) were sent to the facilities for their comment rather than
have SFIP make the call about appropriate linking. Facilities commented on 211 of the 1,790
total permit linkages presented for review. Of these comments, 158 were accepted.
32
December 1999
-------
T bI 2 SFIP A
hI
U.
dP
. L. k
a e : lpproac mproved Sector Dlverses an ermlt ID aees
Universe before Indexing Universe under SFIP
Number of AFS, PCS, Number of AFS, PCS,
Sector Number of RCRIS, and TRI Number of RCRIS, and TRI
Facilities permits/identifiers Facilities permits/identifiers
Petroleum 856 1,877 179 744
Refining
Automobile 1,682 3,465 58 247
Assembly
Pulp 299 603 244 971
Manufacturing
Iron and Steel 550 1,376 114 489
Production
Nonferrous 285 708 44 173
Metal Refining
and Smelting
As mentioned previously, SFIP users from environmental organizations and EP A have
consistently stated that SFIP's externally defined facility "universes" and quality-checked permit
linkages make it a uniquely valuable resource for their analyses. The large number of significant
data integration efforts being undertaken by state agencies suggests that state governments also
place a high value on the ability to assemble accurate facility-level and sector-level profiles. The
project will continue to review the universe of facilities within SFIP to ensure such accuracy.
Additional Findings
In the course of developing complete and accurate sector universes and permit linkages, SFIP has
brought to light a number of issues related to the compliance information maintained in the
underlying data systems. Many of these issues were identified because SFIP effectively focused
attention on data quality assurance and enabled public access (and review) ofthe federal data
systems. Other issues were identified because SFIP also provides, for the first time, a tool that
lets users easily analyze facility-level data. All issues have been resolved or are currently being
addressed. In addition to providing greater public access and improving multimedia facility
profiling, SFIP has also brought about improvements in the management of compliance
information in the underlying databases, such as: guidance for data entry, data correction
procedures, and processing of data as it flows from localities to the federal systems.
.
The Regions have reported that SFIP has assisted them in obtaining cooperation from the
states in responding to data correction requests and improving data management procedures,
resulting in improved data quality.
.
SFIP has identified instances where the current facility status was not known or was not
entered into the data system, including:
33
December 1999
-------
-Was a RCRA permit for a large quantity generator?
- Was a RCRA permit for a Treatment Storage/Disposal facility?
- W as the Significant Noncompliance flag under the Clean Air Act program appropriate?
-Was the return-to-compliance date for RCRA violations entered?
All such issues identified by SFIP in the underlying data bases have been corrected.
However, SFIP has shown the potential value of having an automatic or procedural prompt
that would require staff to revisit or review such status information.
.
Permit identification information (e.g., address, name) was either not collected or not
entered into the Clean Water Act program's Permit Compliance System (PCS). Users
require such information, for example, to distinguish among multiple permits in the same
program. Such information has been collected and entered into the underlying databases as
part of the EP A Regional review of the SFIP data or as facility comments were received.
.
Two EP A Regions had not applied consistent coding protocols/screens for Administrative
Orders (AOs), a type of enforcement action. In one case, certain administrative activities
reported by the state environmental agency, such as '"construction permit received," were
entered into the system as enforcement actions. Such data inaccuracies are now being
screened by the Region prior to submission to the federal database. In another case, actions
not considered AOs at the federal level were being entered as such into the federal database.
This issue is being addressed currently.
.
SFIP identified weaknesses in the processing of compliance data from the initial
statelRegional data systems into the federal systems and ultimately into the Office of
Compliance Integrated Data for Enforcement Analysis (IDEA) system. For example,
processing of AIRS Facility Subsystem (AFS) data has resulted in duplication of a limited
number of enforcement actions under Clean Air Act programs. In another example, the
procedures used to extract Emergency Response Notification System (ERNS) records for
IDEA resulted in the misprinting of material spilled names. Programming changes at the
Region and within IDEA remedied these problems upon discovery by SFIP.
34
December 1999
-------
Assessing Data Quality in Federal Data Systems
Findings concerning the quality of the data compiled by SFIP are divided into two sub-sections.
The first describes the results of an initial Data Quality Assurance Review performed before the
website's launch, while the second describes SFIP's procedures for updating the website data and
maintaining its quality.
In the fall of 1996, EP A invited each state environmental commissioner to provide comments
and assistance on the project, while also providing the states the opportunity to review the data
during a combined EP A Regional/state review period conducted from November 1996 through
February 1997. Prior to SFIP, EP A had never before done this level of research regarding the
multimedia records of individual facilities. The states and EP A Regions worked closely together
during this process. Because most of these data are gathered and maintained by the states before
being compiled in EP A's databases, it was imperative that states be actively involved in this
effort to verify and correct the data. In addition, EP A staff reviewed data maintained by the
Regions which have not been delegated to the states. The Agency regarded this collaboration as
an important process to ensure that the facility-specific compliance and enforcement data are
accurately reported to the public. Most of the comments raised by the states participating in the
review were relatively minor and were resolved through the coordinated EPA-state effort.
During the period surrounding the May 1997 public meeting announcing SFIP, individual
facilities and several trade associations urged the Agency to provide the affected facilities with an
opportunity to review the underlying compliance and enforcement data before the data were
published. In response to these requests, EP A provided each affected facility with copies of its
SFIP data reports in August 1997. Comments received from facilities participating in this review
were addressed prior to the website's launch in May 1998. A similar review process was
followed for 19 "new" or additional facilities identified during a recent review of the SFIP
unIverse.
Since the website's launch, EPA has updated SFIP data from the underlying data systems once
every three months. This keeps the permit-level information current and allows for any
corrections made to underlying data systems to be reflected within SFIP. The SFIP Hotline and
the website's direct emaillink allow facilities and other users to continuously submit comments
and receive immediate direction regarding correction submittal procedures. Because SFIP
reflects unaltered data from several underlying databases, corrections must be made by staff
responsible for maintaining these databases. Designated EP A Regional contacts therefore often
provide assistance in resolving correction requests. These Regional contacts in turn help a
facility identify staff at the Regional or state office who can determine whether the proposed
correction is warranted. Facilities whose corrections are accepted may post comments to SFIP's
Facility Comment Log. As mentioned above, EP A has also reviewed the SFIP facility universe
twice since the website's launch, identifying 19 facilities to be added to the SFIP database and 30
which no longer fall within the defined scope of the project.
35
December 1999
-------
FINDINGS:
SFIP's Data Quality Assurance Review, the most comprehensive quality assurance/quality
control (QAlQC) procedure undertaken to date which included facility review, showed that
compliance data integrated by SFIP are of high quality.
SFIP's commitment to maintaining the highest possible quality of data and intense investment
in processing and review of the data is regarded as one of the project's most valuable aspects
by outside organizations.
Data Quality Assurance Review and Assessment of Data Quality
During the facility review which followed the combined Regional/state review, 62% of facilities
included in SFIP submitted comments. The review categorized data elements into two categories:
"major" elements, which include linked permits, enforcement actions and facility compliance
status; and "minor" elements, which include facility name, address and date of inspection.
Facilities returned comments on approximately 10% of the 37,000 major data elements and
approximately 5% of the 19,000 minor elements that they reviewed. About half of these
comments were accepted by EP A and the states. Thus, the SFIP data reviewed by facilities had a
baseline accuracy of 96%. Specific findings included:
Enforcement Actions => Facilities commented on 64 of the 376 enforcement actions
presented. Of these comments, 41 were accepted. Facilities identified another 20 actions not
listed, ten of which have been accepted.
Significant Noncompliance (SNC) Status => Facilities commented on 103 of the 1,292
eligible data elements. Of these comments, 90 were accepted.
Quarterly Compliance Status => Facilities commented on approximately 3,000 of the 30,000
quarters of historical compliance status presented for review. Of those, comments were
accepted on approximately 2,400 quarters. Note: These values represent an average of the
number of quarters presented and commented upon since comments were not coded
separately for all eight quarters.
Inspections => Facilities commented on 75 of the 3,761 inspections presented. Of these
comments, 31 were accepted. Another 241 inspections were identified, of which 88 have
been accepted.
Of 19 "new" or additional facilities which were sent their compliance data for review in 1999
prior to their formal inclusion in the project, 14 (74%) responded. These facilities submitted
comments on approximately 4% of all major data elements presented, and only 1 % of all minor
elements.
EP A headquarters and Regional staff consider SFIP data to be of high quality, pointing out that
facilities commented on only a "small amount" of the data they reviewed. Some commenters felt
that the initial data quality review demonstrated that such a comprehensive review would not be
36
December 1999
-------
necessary for sectors added in the future. Discussion group participants commented that EPA's
data quality has improved as a result of SFIP. EP A staff also stated that SFIP has provided the
"first comprehensive, credible evaluation of data quality" and suggested that SFIP "can be
thought of as a random sample of data from a relatively large number of sources." Also noted
was SFIP's power to identify opportunities for data quality improvements through systemic
programming changes as well as to pinpoint isolated instances of incorrect data. One EP A data
user summarized SFIP's effect on data quality by saying that SFIP's data quality review means it
is possible to "talk about the quality of the data," and to use the data.
In a discussion group, representatives of environmental organizations described SFIP data as "the
best we have", and "one ofthe most complete and accurate databases...found." They particularly
value the data because it is "quality-checked."
Both trade association representatives and participants in the industry discussion group stressed
their concern with the quality of data made public by SFIP. Trade association representatives
noted that EP A data quality has improved as a result of the project's release and the data's
increased accessibility, while participants in the industry discussion group felt that EP A should
strive to meet a standard of 100 percent accuracy for data released to the public. Based on their
experience in reviewing reports for their own facilities, industry discussion group participants
expressed distrust of SFIP's data quality. One participant stated that although compliance data
presented by SFIP contains only a small number of inaccuracies, even one can have a significant
impact on public perception of a facility or company's "image."
Finally, state agency representatives who attended a discussion group concurred that, because of
the data quality review process, the data in SFIP are probably the most accurate that exist.
Quality Assurance Procedures for Updating SFIP Data
In order to continuously provide up-to-date information to the public, SFIP data is updated, or
"refreshed," quarterly. Since the website's launch in May 1998, SFIP data has been refreshed
five times. Input from all types of stakeholders indicates that users are satisfied with the number
of refreshes being carried out and do not believe more frequent updates are necessary.
Since the website's launch, the SFIP Hotline has processed data correction requests from 72 SFIP
facilities. Of these, eight facilities were concerned only with proposed changes to data which
they had submitted previously, while 51 facilities submitted "new" comments. Thirteen facilities
submitted comments on production data, which they were specifically invited to review in July
1998 and July 1999. In every case, SFIP Hotline staff resolved the problem or referred
commenters to a Regional contact who could help them or direct them to a state data steward.
EP A staff noted that SFIP has resulted in increased attention to data quality within EP A. In the
discussion group, they also credited the project with helping to create greater awareness ofthe
limitations of the current mechanisms for data correction for all EP A databases, noting that there
is "no real identified path for this data correction process."
37
December 1999
-------
Industry representatives agreed that it was time-consuming and "resource-intensive" to monitor
their facilities' data and submit corrections. Industry has objected that the onus rests with
facilities to correct their data, and reported mixed results from their efforts. While several
commenters reported that incorrect data for their facilities remains unchanged, another stated that
"we did have a good experience getting out data corrected, although it took a while." In addition,
one participant commented that being referred back to the ultimate data steward was a difficult
process. Facilities requesting corrections since the site's launch via the SFIP Hotline have also
expressed frustration with having to make additional phone calls and a few also commented that
EP A should bear a greater share of the burden in seeing that corrections are made.
Trade association representatives who met with EP A in March 1999 also relayed a mixed
message regarding their members' experience with submitting proposed data corrections.
Several participants stressed that monitoring the quality of their data takes staff time, which is a
scarce resource for some facilities. While many facilities have had data correction requests
resolved in a timely manner without difficulty, some facilities have indicated enough frustration
with the process that they had stopped trying to correct data errors. Some pulp manufacturers
have expressed frustration with being referred by the SFIP Hotline to multiple data stewards.
Participants suggested that EP A appoint an "ombudsman" for facility data correction requests,
who would provide one point of contact for all data issues and be able to update facilities on the
status of their requests.
38
December 1999
-------
Future Directions
SFIP has provided the Agency with an important new tool with which to re-orient the many
activities related to data analysis along industry sector lines. This project evaluation has shown
that users believe that in order to continue the contributions SFIP has made to date, the project
must shift from being a pilot to a more comprehensive sector-based analysis and tracking tool
that is available to both internal and external users. To that end, the tool must be expanded to
include additional sectors. Users recommended numerous sectors for SFIP's expansion. The list
of recommended sectors which are both feasible to include and which are Agency priorities
includes but is not limited to: metal services, electric utilities, paper manufacture, and federal
facilities. Information on these and the other sectors recommended for addition to SFIP are
found below.
In addition to expanding the number of industry sectors included in SFIP, the evaluation revealed
that users want and would benefit from other changes to SFIP. Some users recommended
changing the website's functionality (changing what can be done with the site and/or SFIP
information), for instance, by allowing queries on a user-selected set of facilities or providing the
ability to do time-series analyses. Others wanted additional information added into the SFIP tool
or to have certain information more readily linked to the site.
The most frequently recommended changes include:
1.
Adding new sectors, including but not limited to: metal services, electric utilities, paper
manufacture, and federal facilities.
The procedures to add sectors have been established and have been proven to yield high-
quality information. The findings of SFIP' s initial data quality assessment indicate that it
may not be necessary to repeat such a comprehensive and costly review for new sectors.
Resource requirements for adding sectors are associated with sector characteristics, such
as number of facilities, complexity of operations, and typical permit requirements.
Adding new data elements to information already provided, including, for example:
pollutant releases and waste management information from CW A, CAA and RCRA; toxicity
weighting; OSHA inspections and more details regarding permits (e.g., expiration dates and
emissions limits)
Adding information that is maintained within the currently-linked data systems can be
done with relatively little difficulty. Establishing links to other data systems or entering
data from outside sources would be more resource intensive.
Extending compliance information beyond the most recent two years
The underlying data systems do not currently maintain the relevant data for more than
two years. Therefore these underlying systems would have to maintain the information
for a longer period before such a change could be considered for SFIP. Alternatively,
SFIP could make the archived SFIP data (the facility and sector level statistics, now
stored in spreadsheet form) directly queriable.
Providing users the ability to do geographic mapping and/or to conduct cross-sector analysis
in a geographic area
Research to determine the feasibility of this suggestion could include the development of
a prototype that would extend the SFIP tool to cover all facilities in a given geographic
2.
3.
4.
39
December 1999
-------
5
area, such as a metropolitan area (and would include numerous smaller facilities) This
would allow EP A to pretest the value of Geographic Information Systems (GIS) query
capabilities This effort might serve as a useful pilot for the GIS tool that is currently under
development by OC
Giving users the ability to conduct time series analyses
Adding this function would require additional investigation into types of time-series
analyses users would need Time series would not be meaningful using the two-year time
period currently maintained in SFIP (See above comment regarding feasibility of adding
older data)
These and other recommended changes are currently being evaluated for feasibility as part of the
SFIP planning process
Additional sectors
Throughout the first year since the SFIP website launch as well as during this evaluation process,
all types of commenters have recommended that SFIP should expand to include more industry
sectors Many see the fact that SFIP provides data for only five of the hundreds of major
industrial sectors as the project's biggest limitation The majority of specific requests for
additional sectors have come ITom EP A Regional staff, but every group with which the project
had interaction has submitted suggestions Regional Office staff, state governments, public
interest and environmental groups, facility representatives, trade associations and individual
citizens While environmental groups, EP A and states wanted to be able to access data for
additional sectors, industry felt that SFIP unfairly "spotlights" the included sectors, implying that
they are among the worst in terms of environmental performance F or this reason, industry
representatives were interested in seeing SFIP cover all sectors
As with all comments that have been provided to EPA throughout the project's first year of
availability, each recommendation concerning the expansion of SFIP is being given serious
consideration While EPA has not made a final determination on how best to proceed with SFIP,
the Agency has begun to study both the feasibility and appropriateness of adding the sectors that
have been widely suggested EP A is considering a variety of factors in addition to information
gathered through the evaluation process documented in this report The Agency's initial
assessment for each sector suggested by SFIP users is provided below Sectors which are listed
under the heading "Tier I" have been identified as such based on whether the sector is a current
priority of the Agency, the feasibility of obtaining permit and facility information for the sector,
and the number of requests to add the sector that have been received
Tier I Sectors:
Metal services - Metal Services is a National Enforcement Priority Sector for FY2000 OC is
currently leading the Agency's efforts in working with the sector to provide compliance
assistance Several Regions have mentioned chrome plating as a priority sector for which
SFIP expansion would be a valuable tool On the other hand, there are tens of thousands of
metal finishing operations in the US, most of which are small and do not hold federally
reportable permits Many are captive operations in plants that make, for example, appliances
These characteristics make facilities in this sector more difficult to compare and somewhat
problematic to incorporate into SFIP
40
December 1999
-------
Paper manufacture - Information on this sector would complement the data for the upstream
sector of pulp manufacture, a current SFIP sector. Permit linkages for many of the stand-
alone paper manufacturing facilities have already been developed and sources for production
information have been identified.
Electric utilities - Large emitters of both criteria air pollutants and toxic air pollutants. The
sector is now subject to TRI reporting for 1997 and beyond. In addition, the Natural
Resources Defense Council and the Clean Energy Group, an industry association of utilities,
recently committed itself to and called for increased disclosure of toxics emissions from all
u.s. generating operations. Within the sector there are several logical subsectors, such as
fuel type (e.g., coal, oil, gas, nuclear, solar).
Tier II Sectors:
Airline operations - Information on stationary sources of pollution (e.g., cleaning, repainting) is
amenable to such indexing, but the operations of the mobile sources (aircraft) are not.
Automobile repair - Such facilities are numerous and nationally distributed. They pose
environmental concerns related to hazardous waste generation, sewer discharges and threats
to groundwater. OC currently supports compliance assistance efforts within the sector and
there are several state and many local authorities now collecting performance information
from facilities in this sector. However, facility and permit information maintained within the
federal data systems is very limited.
Automobile manufacturing sectors other than assembly (e.g., manufacture of components) - This
sector should be limited to parts or component manufacturers that supply the automobile
industry .
Biologicals - Facilities tend to be concentrated in certain geographic areas. While there is
currently a great deal of research and development activity in this sector, more emissions
and regulatory requirements will be triggered when plants move into full production.
Car washes - Like automobile repair facilities, car wash facilities are numerous and nationally
distributed. Environmental concern about this sector is related to sewered discharges and
threats to groundwater. However, facility and permit information maintained within the
federal data systems is very limited.
Chemical fertilizer production - OC currently has an internal tool to provide compliance
information for chemical manufacturers. Information already collected for chemical
fertilizer producers could be built upon if the sector is added to SFIP.
Concentrated animal feed operations (CAFOs) - A current Agency priority. Ongoing initiatives
are expanding the regulatory and reporting requirements for such operations. However, the
federal data systems contain limited information on their past performance. This sector is a
good candidate for a future addition, once the information for a large fraction of such
facilities is included in the federal data systems.
Consumer goods - This sector is composed of facilities involved in the production of a wide
variety of products. Since this is a diverse industry, it would be necessary to develop
appropriate subsectors.
Electronics manufacture - Like the consumer goods sector, this is a diverse industry for which it
would be necessary to develop appropriate subsectors. Printed Wiring Board Manufacture
(currently in partnership with the Agency's Design for the Environment Program) and
Integrated Circuit Manufacture are two potential subsectors.
41
December 1999
-------
Federal facilities - Universe would likely include a subset of the operational sites (either a type of
facility and/or facilities within a particular Department or Agency). Nominations were also
received for including closed military bases/sites. This sector is the focus of the Federal
Facility Enforcement Office within OECA. Including federal facilities in SFIP would
provide the public with additional information on the U.S. government's own performance.
Foundries - Selection of stand-alone operations would increase the comparative value of the
facility-level information.
Hazardous materials/waste treatment - RCRA subtitle C facilities and solvent recyclers are now
subject to TRI reporting for 1997 and beyond.
Major sources subject to maximum achievable control technology (MACT) standards - These
facilities tend to be large and fairly complex, with permits in several regulatory programs.
Thus indexing across programs has the potential to significantly improve facility
information. For inclusion of this sector to be feasible, it would be necessary to limit the
universe to a practical number of facilities.
Mining - Another diverse industry for which it would be necessary to develop appropriate
subsectors, e.g., coal mining, strip mining, metal mining, or hard rock mining). In
particular, adding a metal mining sector would complement the current SFIP nonferrous
metal refining and smelting sector. Metal mining and coal mining are now subject to TRI
reporting for 1997 and beyond.
Municipal waste combustors - Of environmental concern because they are typically located in or
near metropolitan areas with significant releases to air. It may be better to define this sector
strictly by operation type rather than by ownership, and thus include all waste combustion
and/or cogeneration facilities.
Oil and gas production - Both land and off-shore operations could potentially be included within
this sector. The list of currently active sites may be difficult to develop and maintain
because production facilities shift between idle and operational depending upon oil prices.
Pharmaceuticals - Facilities tend to be stand-alone operations and are concentrated in certain
geographic areas. However a potential difficulty is that this sector includes facilities where
R&D operations are mixed with full production operations.
Secondary nonferrous metals - Inclusion of this sector would provide a useful comparison with
data for the primary nonferrous metals sector, which is currently included in SFIP.
Sectors covered by OC's Industry Sector Notebooks - The Sector Notebooks were developed for
OC priority sectors and/or for sectors for which sector-level information was lacking. All
five of the current SFIP sectors are the subject of Sector Notebooks. Users have suggested
that expanding SFIP to include the remaining Sector Notebook sectors would be particularly
useful.
Wood products - A well-defined facility universe and permit linkages for many of the pressed-
wood products facilities have already been developed for this sector.
Sectors covered by FY2000 MOAs - Petroleum Refining is a current SFIP sector. Metal Services
is listed under Tier I.
Alternative/Additional Functionality of the SFIP website
Suggestions include:
.
Create a version of SFIP that provides whole-facility data for all facilities in a given
geographic area (see above).
42
December 1999
-------
.
Provide a direct link from Facility-level Statistics Reports to the Aggregate Summary
Statistics Page for the whole sector.
Provide links to state environmental websites.
Provide contact numbers for users to obtain additional background on program-specific data,
i.e., the TRl hotline.
Provide additional compliance record details for the included facilities. This could be done
by expanding the existing SFIP Detailed Facility Report, by increasing SFIP's connectivity
to IDEA, or by linking SFIP's permits to EPA's Envirofacts System.
Make it possible to access detailed data and summary statistics for a group of facilities with
one search request.
.
.
.
.
Additional Information
Suggestions include:
.
Add informal enforcement actions, such as Notices of Violation (NOVs).
Replace production capacity with actual production where possible (currently this data is
only available for automobile assembly plants).
Add information on Supplemental Environmental Projects (SEPs) and other civil docket
information (useful for environmental justice goals).
Include state or EP A docket number for Enforcement Actions.
Add penalty comment field.
Include information on the quality of the air (degree to which area has met National Ambient
Air Quality Standards for the six criteria air pollutants: SOx, NOx, CO, particulate matter,
ground-level ozone, lead) surrounding the facility.
Expand SFIP to create a one-stop comprehensive resource for information on included
sectors. Users have suggested that the SFIP website could serve as a clearinghouse for
numerous kinds of sector-specific information (e.g., history of the industry, etc.) .
Include information about compliance assistance activities.
Add voluntary facility initiatives with positive environmental effects to the facility profiles.
Add hypertext links to company Environmental Health and Safety (EHS) managers and to
company home pages "for the other side of the story."
Expand the compliance history beyond two years to five years (see above).
Include ownership/corporate-level data.
Include the cause of pollutant spills.
Include EPCRA 311, 312 Chemical Storage and Emergency Response Plans.
Include detailed pollutant reports, including:
- Clean Air Act: criteria air pollutants; dominant air pollution emission source
- Clean Water Act: toxic and conventional discharges
- Resource Conservation and Recovery Act: manifested waste information
- Toxics Release Inventory: chemical-specific release and transfers
Add toxicity weighting for use with Toxics Release Inventory reported releases.
Add sector-level economic measures, such as number of employees, and percent
contribution of sector to Gross National Product.
Include maps of the surrounding area showing other pollutant sources, population and
important environmental resources.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
December 1999
-------
APPENDIX 1
WEBSITE USE STATISTICS REPORTS
-------
Cumulative Use Statistics for
the Sector Facility Indexing Project (SFIP)
Reporting on the Initial 10 Weeks:
May 1, 1998 through July 11, 1998
-------
The Sector Facility Indexing Project (SFIP) provides greater public access to environmental data
for approximately 650 facilities in five industry sectors. As the SFIP is a pilot project, the
Agency plans to evaluate it over the next few months to determine future directions and before
making a decision on any significant expansion. As part of this evaluation, we are tracking the
usage rates (user sessions and hits) of the website, and identifying trends in how the site is being
accessed, and to the extent possible, what type of organizations are using the site. The following
report contains our initial findings for how the SFIP has been accessed from its release through
July 11, 1998. Such statistics will be gathered on a regular basis. A second report in October
will include an assessment of how the data refresh of August 13 affected website use.
In addition to the statistical reports, we plan to evaluate public awareness of the project, and
customer satisfaction with the information provided; assess the utility of The SFIP as a
compliance and analytical tool; and conduct focus groups to gain in-depth knowledge of how the
project's information is being used and how we can improve/expand the SFIP. We plan to
initiate these activities in November 1998, six months after the initial release of the data. This
will allow sufficient time for us to promote the SFIP, and for users to become familiar with the
website and its various functions. This also will allow us to refresh the data two times after the
initial release (in addition to the August 13 refresh, we plan to complete a second refresh in early
October).
Through such an evaluation, we will be able to make a qualitative assessment of the project and
whether we are achieving our goals for establishing the SFIP.
SFIP Goal:
SFIP Goal:
Assessment:
SFIP Goal:
Assessment:
Improve multimedia facility profiling and sector-based analysis.
Develop analytical tool for government to identify compliance patterns and
determine where to place scarce resources.
There are currently five OECA projects using the SFIP for root cause analysis,
TRI reporting targeting and sector strategy efforts.
Provide greater public access to compliance and facility-level information.
In reviewing the statistics that we have already gathered, we believe we are
presently meeting the goal of being a resource used by the public. Throughout the
first two months following its release, the SFIP Website has received a consistent
level of interest from a wide range of public users as well as staff at all levels of
government. Furthermore, feedback about the SFIP from e-mail, the Hotline and
letters, shows that this effort to make public readily understandable environmental
performance information (including compliance and enforcement records) is
providing reliable and understandable information. There are relatively few
comments regarding erroneous information on the site -- each is investigated as
soon as we are notified. Very few users reported having difficulty using the site--
in most cases, they wanted to (and could not) access information on facilities
beyond the scope of the SFIP.
SFIP Cumulative Use Statistics through July 11
1
-------
Assessment:
Provide industry with increased ability to design self-policing and compliance
assistance programs.
Insufficient time has passed since the SFIP launch to characterize industry
responses. Identifying and characterizing industry responses will be the focus of
future evaluation work.
SFIP Goal:
This memorandum provides statistics on the website's use and summarizes comments received
through the SFIP Hotline, via the site's e-mail link, and by regular mail.
SFIP Cumulative Use Statistics through July 11
2
-------
Findings
.
The long anticipated release of the SFIP led to a high degree of use in the fIrst week of
the site's availability. The number of user sessions! and hits2 were at their peak
immediately following the initial launch.
.
User sessions and hits have leveled off since the post-launch peak, when over 6,000
users accessed the site in the fIrst week. Since then, usage has leveled out to about
1,700 users per week. This leveling off was expected and is considered normal. (It
should also be noted that the site was launched just prior to the summer season when
fewer people work.)
.
Data use is the primary activity of the site's users. On-line analysis, downloading of
data fIles and review of facility records account for between 58 and 67 percent of total
hits. While users review the other pages for needed information, they then do what we
had hoped - access the data.
.
Data for the Petroleum RefIning sector and its facilities received the greatest amount of
attention. In contrast, while pulp manufacturing data was accessed less often than
expected given the sector's size, pulp manufacturers such as Champion Paper,
International Paper, Weyerhaeuser, and Georgia PacifIc were among the site's most
frequent users. Other frequent users of the site included EP A staff at the Regions and
HQ, petroleum refIning companies such as Amoco, Exxon, and Citgo, and a steel
manufacturer.
.
Views of the Facility Comment log accounted for one percent of total hits each week,
making it the 12th most viewed page. Users are clearly interested in facility comments
on their data.
.
108 public comments on the site had been received as of July 11, 1998, approximately
half of which came from SFIP facilities an~ trade associations. The largest number of
comments came from the petroleum refIning sector, followed closely by the pulp
manufacturing sector.
site.
! A "user session" is a discrete period of activity generated by a unique user.
2 A "hit" is an action on the Website, such as the viewing of a page or the downloading of a file from the
SFIP Cumulative Use Statistics through July 11
3
-------
High Degree of Use Immediately Following Release
SFIP Web Use Trends' User Sessions (Left Axis) and Total Hits (Right Axis)
7,000
50,000
1,000
40,000
6,000
5,000
en
c:
.Q 4 000
m '
Cf)
'"- 3,000
Q)
~ 2,000
30,000
~
I
20,000
10,000
o
o
. User Sessions
-{)- Total Hits
April 26 rv"ey 1 0 May 24* June 7 June 21 July 5
rv"ey 3 rv"ey 17 rv"ey 31 June 14 June 28
Week beginning
*included Memorial Day holiday
Between May 1 and July 11, users logged a total of23, 167 sessions on the site The site received
135,674 hits in this period
Use of the site was most intense during the first full week after the Friday, May 1 launch In that
week, there were 6,178 user sessions, and 44,166 hits Use was fairly consistent over the
subsequent 9 weeks, with an average of 1,714 user sessions per week and 9,019 hits per week
Consistent usage indicates that there is a continual stream of new users (that replace those who no
longer access the site) Such expanding publicity and a growing number of individuals who are
aware of the site are important elements for broadcasting the existence and uses of the SFIP
An increase in both hits and user sessions in the week beginning July 5 is most likely attributable
to the posting of several new pages presenting 1996 facility production data in preparation for the
first data refresh The recent mailing of an SFIP press package to media across the country may
also have contributed to the increased activity on the site
SFIP Cumulative Use Statistics through July 11
4
-------
Data Use Accounted for 57-64 percent of Activity on the Website
Proportion of Hits related to SFIP Data Use
April 26 May 10 May 24* June 7 June 21 July 5
May 3 May 17 May 31 June 14 June 28
Week Beginning
"included Memorial Day
A majority of the total hits to the site were generated by users accessing the SFIP's data
A conservative estimate of data use (the lower bound) was estimated by combining the following
tallies
• Page views3 directly related to data appearing on the list of 50 Most Requested Pages
(includes sector Data Summary pages, sector Data Access pages, Search pages, etc )
• Data downloads
• Forms and Scripts4 submitted by users
The upper bound estimate of data use on the Website was generated by adding views of pages
which were not accessed frequently enough to appear on weekly lists of "50 Most Requested
Pages " Because all major pages (the Home Page, Data Access, Indicators, and search pages,
etc ) are represented in the "Most Requested" list, these "unknown views" are assumed to be
mostly detailed facility reports views
A "page view" is a hit to an html text page this tally excludes downloads, and forms and scripts (see
below)
4 "Forms and scripts" are used to perform searches of SFIP data For example, a user might input a
facility name or state into a form on one of the Easy Search pages Clicking on "Submit" activates a script that
searches for the desired data
SFIP Cumulative Use Statistics through July 11
5
-------
Petroleum Sector Received the Greatest Amount of Attention on the SFIP Website
Number of
Cumulative Facilities in
Pa2e Viewed Hits Sector IlitslFacility
Petroleum Refinin~ Data Access 3,165 179 18
Pulp Manufacturers Data Access 2,317 247 9
Iron and Steel Mills Data Access 1,536 118 13
Primary Nonferrous Metals Data Access 1,326 51 26
Automobile Assembly Data Access 1,230 58 21
The Petroleum Refining sector received the highest number of total hits (3,165), but the Primary
Nonferrous Metals and Automobile Assembly sectors had higher ratios of hits per facility.
Although the Pulp Manufacturing Sector has the largest number of facilities (247), it received the
second highest number of hits, and had the lowest ratio of hits per facility (9).
Prior to the next phase of SFIP evaluation (during which facilities and other users will be
interviewed), we do not know the specific reasons for differences in usage across sectors.
However, the following list includes several different factors which affect the measured sector-
specific usage:
.
Activity of the industry trade associations in reviewing the SFIP.
SFIP facilities' awareness of Website and activity reviewing information.
The number of facilities in the sector (e.g., if users view each sector equally, there will be
more usage per facility in sectors with fewer facilities).
Whether the industry is an EPA priority (a major user).
Interests of the general public.
.
.
.
.
SFIP Cumulative Use Statistics through July 11
6
-------
Frequent Users
User
Identifiable Or2anizationffypes Total Hits Sessions
EPA 5,740 898
Pulp Manufacturing sector 2,779 157
Petroleum Refining sector 888 42
Research Institutions 860 39
Other Federal Agencies 237 8
Other Industry sectors 143 7
Steel Manufacturill1Z sector 135 8
10.782 1.159
Using weekly lists of the 10 "Most Active Organizations", approximately 17 percent (3,938) of the
23,167 user sessions logged from May 1 to July 11 are identifiable by their Internet domain names.
Of these, about 70 percent originated from the domain names of Internet service providers (ISPs),
such as America Online or Erols, and no further identification of users can be made.
For 1,159 user sessions, or 5 percent of the total, users can be identified by organization name.
These users are listed in the chart, ranked by total hits. However, because statistics are available
only for the 10 most active organizations in a given week, these rankings are only approximate, and
each organization type is likely to have logged more user sessions and hits to the site than are
shown here.
SFIP Cumulative Use Statistics through July 11
7
-------
Slight Decline in Hits per Users Session as Expected
I Hits per User Session I
8
.~ 6
en
en
Q)
Cf) 4
....
Q)
C-
en
~ 2
o
April 26 May 10 May 24* June 7 June 21 July 5
May 3 May 17 May 31 June 14 June 28
Week beginning
"included Memorial Day holiday
A declining ratio of hits per user session is expected as repeat users become more familiar with the
website and use shortcuts to access only the information they wish to retrieve. For example, users
who have previously read SFIP data element descriptions may not need to consult the Indicators
page again. Users who are interested only in a particular sector may have bookmarked that sector's
Data Access page, bypassing the home page a nd the general Data Access page.
SFIP Cumulative Use Statistics through July 11
8
-------
The Facility Comment Log Consistently Received Attention
During each week since the site's launch, from May I to July 11, the Facility Comment Log
received one (1) percent of all hits by users. Cumulatively, the Facility Comment Log was the
12th most viewed page.
Thus, we can see that users are interested in knowing whether facilities have needed to respond
to the data that we have released.
Five facilities submitted comments for posting on the site.
SFIP Cumulative Use Statistics through July 11
9
-------
Summary of Public Comments as of July 11, 1998
A total of 108 inquiries and comments had been received as of July 11. These included:
. 70 Hotline calls (from 60 different facilities or organizations)
. 32 e-mails
. 6 written letters
Facilities or their corporate representatives accounted for 54 of 108 public comments (50%),
including:
.
20 from the petroleum refining sector
19 from the pulp manufacturing sector
11 from the primary nonferrous smelting and refining sector
3 from the iron and steel production sector and
1 automobile assembly facility (BMW)
.
.
.
Sixty-one of 108 public comments (56%) concerned the data presented on the Website. Of these,
. 28 facilities or their corporate representatives requested corrections for their data. Of
these,
15 were new requests
11 concerned data for which corrections had previously been requested but not
made by Regions or States before the August refresh.
- 1 facility volunteered information about a change of facility ownership and
- 1 facility did not provide details before being referred to the Regional contact
16 comments included requests for explanations of particular data elements.
Most frequent comments related to DMR filings, including disputed violations and
late filings as well as nonexistent permit limits. These were investigated as received
and issues that could not be resolved by contractor and SFIP staff were forwarded to
the appropriate Regional and staff.
Additional comments included:
. 20 questions about how to access or use the site, all of which were easily and
immediately addressed.
. 11 requests for more information about the project
. 5 inquiries or suggestions about sectors to be added in the future
One user suggested adding the large-scale animal feeding industry, while another felt
that hazardous materials treatment facilities "should be at the top of your list".
. 3 users offered congratulations on the site's launch, while
. 2 users expressed concern that the site is hard to fmd or to use. One user expressed a
wish for clearer signposts to help users get to the SFIP from the EP A main home
page. The second user may have had problems with their software but could not be
contacted (leaving no return address).
SFIP Cumulative Use Statistics through July 11
10
-------
Project Use by EPA:
To date, there are currently five OECA projects using the SFIP for a variety of purposes:
. 1998 Inspection Targeting for RCRA Enforcement Section (Office=Region IX)
. Nonferrous Metals Root Cause Analysis (Office=OC/METD)
. Iron and Steel Sector Strategy (Office=ORE, Region V)
. Iron and Steel Root Cause Analysis (Office=OC/METD)
. TRI Sector Targeting (Office=OCIEPTDD)
SAP Cumulative Use Statistics through July 11
11
-------
Cumulative Use Profile for
the Sector Facility Indexing Project (SFIP)
I Update #2 I
May 1 through October 3, 1998
-------
This memorandum provides statistics on the website's use and summarizes comments received
through the SFIP Hotline, via the site's e-mail link, and by regular mail. In considering these
statistics, it may be helpful to keep in mind the following significant events in the life of the site:
8m: T JIUIIchl"Jll
May I, 1998
1996 productiou data
posted for facility
rmcw July 3
Fccknl Register notice
8IIDOUDces availability of
Propss Report
t!
>
~
t-~
~
~
IDitial SFlP
Use Profile
1hrough July 12
DIda R.efrcshcd
August 13, 1998
This updaIc:
tbrougb October 3
Findings
.
User sessions and hits have remained steady since the launch, averaging approximately
1,500 users per week.
.
Data use continues to be the primary activity of the site's users. On-line analysis,
downloading of data files and review of facility records accounted for between 50 and
67 percent of total hits.
.
The Petroleum Refining sector continues to receive the greatest amount of attention,
while the Primary Nonferrous Metals Smelting and Refining and Auto Assembly
sectors continued to have the highest ratios of hits per facility.
.
Pulp manufacturers and petroleum refining companies continued to be among the most
frequent users of the site. Frequent new users include the Environmental Defense Fund
and the Environmental Working Group, as well as two states, Missouri and Maine.
.
The Facility Comment log was the 11 th most viewed, indicating strong user interest in
whether facilities have submitted corrections for their data.
.
Of 192 inquires or comments received since the site's launch, 84 were received
between July 12 and October 3, 1998. Almost half of these were requests for copies of
the SFIP Progress Report. Approximately 36% (30) came from SFIP facilities and
trade associations, 19 of which were from the Petroleum Refining sector. Commenters
were more likely to request explanations and less likely to request data corrections than
previously.
SFIP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 1
-------
BacklITound
The Sector Facility Indexing Project (SFIP) provides greater public access to environmental data
for approximately 650 facilities in five industry sectors. As the SFIP is a pilot project, the
Agency plans to evaluate it over the next few months to determine future directions and before
making a decision on any significant expansion. As part of this evaluation, we are tracking the
usage rates (e.g., user sessions and hits) of the website, and identifying trends in how the site is
being accessed, and to the extent possible, what type of organizations are using the site. This, the
second report, contains our findings on how the SFIP has been accessed during its first five
months. Use statistics will continue to be gathered on a regular basis.
In addition to the statistical reports, we plan to evaluate public awareness of the project, and
customer satisfaction with the information the SFIP provides; assess the utility of the SFIP as a
compliance and analytical tool; and conduct focus groups to gain in-depth knowledge of how the
project's information is being used and how we can improve/expand the SFIP. We plan to
initiate these activities in January 1999, eight months after the initial release of the data. This
schedule will allow time for users to become familiar with the website and its various functions.
The evaluation will enable us to make a qualitative assessment of the project and whether we are
achieving our goals for establishing the SFIP.
SFIP Goal:
SFIP Goal:
Assessment:
SFIP Goal:
Assessment:
Improve multimedia facility profiling and sector-based analysis.
Develop analytical tool for government to identify compliance patterns and
determine where to place scarce resources.
Eight projects using SFIP data have been identified. Six OECA projects are using
the SFIP for root cause analysis, TRI reporting targeting and sector strategy
efforts. Region X's PCB Coordinator has used SFIP inspection data for
enforcement and compliance assistance targeting. Region VI is using the SFIP to
assess the accuracy of Standard Industrial Classification code reporting by
petroleum refineries. In addition, a systematic review of RCRIS data currently
being undertaken by OECA was prompted by the discovery of a data quality
problem (omission of return to compliance information) during development of
the SFIP.
Provide greater public access to compliance and facility-level information.
In reviewing the statistics that we have already gathered, we believe we are
presently meeting the goal of being a resource used by the public. In the first five
months following its release, the SFIP Website has attracted a wide range of
public users as well as staff at all levels of government. Furthermore, feedback
via e-mail, the SFIP Hotline and letters, shows that users feel SFIP is providing
reliable and understandable information. There are relatively few comments
regarding erroneous information on the site -- each is investigated as soon as we
are notified. Very few users have reported having difficulty using the site.
SFIP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 2
-------
SFIP Goal:
Assessment:
Provide industry with increased ability to design self-policing and compliance
assistance programs.
Insufficient time has passed since the SFIP launch to characterize industry
responses. Identifying and characterizing industry responses will be the focus of
future evaluation work.
SAP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 3
-------
Website Use Remains Steady
SFIP Web Use Trends: User Sessions (Left Axis) and Total Hits (Right Axis) I
7,000
6,000
5,000
O 4,000
(/)
to
0)
u_ 3,000
-------
Data Use Continues to Account for More than Half of Activity on the Website
JS
o
"o
o>
o>
"c
0)
I
100
90
80
70
60
50
40
30
20
10
0
I Proportion of Hits related to SFIP Data Use II
r~i Lower bound B Upper bound
April 26 May 10 May 24* June 7 June 21 July 5 July 19 Aug 2 Aug 16 Aug 30 Sept 13 Sept 27
May 3 May 17 May 31 June 14 June 28 July 12 July 26 Aug 9 Aug 23 Sept 6 Sept 20
Week Beginning
In the five months between May 1 and October 3, 1998, between 50 and 67 percent of all hits were
generated by users accessing the SFIP's data1.
The increasing gap between the lower and upper bound probably reflects repeat users accessing
detailed facility reports more frequently (increasing upper bound) and using shorter paths to reach
them (decreasing lower bound).
A conservative estimate of data use (the lower bound) was estimated by combining the following tallies
Page views directly related to data appearing on the list of 50 Most Requested Pages (includes sector Data Summary pages,
sector Data Access pages, Search pages, etc )
• Data downloads
• Forms and Scripts submitted by users
The upper bound estimate of data use on the Website was generated by adding views of pages which were not accessed frequently
enough to appear on weekly lists of "50 Most Requested Pages " Because all major pages (the Home Page, Data Access, Indicators,
and search pages, etc ) are represented in the "Most Requested" list, these "unknown views" are assumed to be mostly views of
detailed facility reports and facility-level statistics reports
SFIP Cumulative Use Statistics through October 3, 1998 - Update 2
PageS
-------
Petroleum Sector Continues to be the Most Viewed Sector
Number of
Cumulative Facilities in
Pa2e Viewed Hits Sector HitslFacility
Petroleum Refining Data Access 4,135 179 23
Pulp Manufacturers Data Access 3,102 247 13
Iron and Steel Mills Data Access 2,050 116 18
Primary Nonferrous Metals Data Access 1,773 51 35
Automobile Assembly Data Access 1,585 58 27
During the five months prior to October 3, users continued to give their attention to the various
sectors in approximately the same proportions as in the site's first three months. The Petroleum
Refining sector continued to receive the highest number of hits (970 for the recent period, 4,135
cumulative). The Primary Nonferrous Metals and Automobile Assembly sectors continued to
have the highest ratios of hits per facility. Although the Pulp Manufacturing Sector has the
largest number of facilities (247), it has received only the second highest total number of hits,
and has the lowest ratio of hits per facility (13).
Prior to the next phase of SFIP evaluation (during which facilities and other users will be
interviewed), we do not know the specific reasons for differences in usage across sectors.
However, the following list includes several different factors which affect the measured sector-
specific usage:
.
Activity of the industry trade associations in promoting their members' awareness of
SFIP;
Level of awareness and interest among a given sector's facilities;
The number of facilities in the sector (e.g., if users view each sector equally, there will be
more usage per facility in sectors with fewer facilities);
Whether the sector is an EP A or state priority (EP A is a major user of the site); and
Interests of the general public.
.
.
.
.
The Facility Comment Log Received Consistent Attention
The Facility Comment Log is the 11 th most viewed page, indicating strong user interest in
whether facilities have submitted corrections for their data.
As of October 3, five (5) facility comments had been posted on the site.
SFIP Cumulative Use Statistics through October 3,1998 - Update 2
Page 6
-------
Frequent Users
User
Identifiable Or2anizationffvve Total Hits Sessions
EPA 10,265 1,579
Pulp Manufacturing sector 3,791 244
Petroleum Refming sector 1,292 90
Research Institutions 946 43
Other Federal Agencies 441 20
Environmental Organizations 210 33
Other Industry sectors 209 9
Steel Manufacturing sector 135 8
States 130 6
Law Office 44 3
TOTAL 17,463 2,035
Using weekly lists of the ten "Most Active Organizations,"2 2,035 user sessions (6% of 33,952),
users can be identified by organization name. However, because statistics are available only for
the ten most active organizations in a given week, these rankings are only approximate and each
organization type identified above is likely to have logged more user sessions and hits to the site
than are shown here.
2 Approximately 21 percent (7,227) of the 33,952 user sessions logged from May 1 to October 3 are identifiable by their Internet
domain names. Of these, about 65% originated from the domain names of Internet service providers (ISPs) or search engines, and no further
information about the individual users accessing SFIP is available.
SFIP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 7
-------
Public Comments, July 12-0ctober 3, 1998
Trends
Of 192 inquires or comments received since the site's launch, 84 were received between July 12
and October 3, 1998.
Almost half (41 of 84) of these were requests for copies of the SFIP Progress Report.
For SFIP facilities, petroleum refmeries continued to be the most frequent commenters. A smaller
proportion of recent commenters were pulp manufacturers and nonferrous smelters/refiners.
Recent comments concerning SFIP data were more frequently requests for explanations, and less
frequently requests for corrections.
Recent requests for data corrections were more likely to be first-time requests; follow-up calls
concerning previously requested corrections were less frequent.
The 84 comments received included:
. 71 Hotline calls (from 63 different facilities or organizations)
. Ten (10) e-mails
. Three (3) written letters
Facilities or their corporate representatives accounted for 30 comments (36%), including:
. 19 from the petroleum refining sector
. Six (6) from the pulp manufacturing sector
. One (1) from the primary nonferrous smelting and refining sector
. Two (2) from the iron and steel production sector and
. Two (2) automobile assembly facilities
Twenty-three public comments (27%) concerned the data presented on the Website. Of these:
. 12 facilities or their corporate representatives requested corrections for their data. In
addition, 2 state agency staff called about facilities who disputed SFIP data. Of these
data correction requests:
- Seven (7) were new requests
- One (1) concerned data for which corrections had previously been requested but not
made by Regions or States before the August refresh.
- Six (6) concerned the 1996 production data which facilities reviewed in July
. Ten (10) comments included requests for explanations of particular data elements.
These were investigated as received and forwarded to Regional staff when necessary.
SFIP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 8
-------
Additional comments included:
.
Seven (7) questions about how to access or use the site, all of which were easily and
immediately addressed.
Nine (9) requests for more information about the project, including three (3) inquiries
about EPA's plans to add other sectors in the future.
Three (3) callers were seeking data not presented by SFIP, including one request each
for "chemical product information", corporate-level environmental performance data,
and statistics on criminal enforcement actions.
.
.
Project Use by EPA:
To date, eight projects using the SFIP have been identified.:
~ 1998 Inspection Targeting for RCRA Enforcement Section (Qffice=Region IX)
~ Iron and Steel Sector Strategy (Office=ORE, Region V)
~ Iron and Steel Root Cause Analysis (Office=OC/MEID)
~ TRI Targeting (Office=OCIEPTDD)
~ Nonferrous Metals Root Cause Analysis (Office=OClMETD)
~ Nonferrous Metals Sector Strategy (Office=OECNOSW)
~ PCB enforcement and compliance assistance targeting by Region 10
~ SIC Quality Assurance for Petroleum Refineries by Region 6
In addition, a systematic review of RCRIS data currently being undertaken by OECA was
prompted by the discovery of a data quality problem (omission of return to compliance
information) during development of the SFIP.
SAP Cumulative Use Statistics through October 3, 1998 - Update 2
Page 9
-------
APPENDIX 2
SAMPLE DISCUSSION QUESTIONS
FROM SFIP FOCUS GROUPS
-------
SAMPLE DISCUSSION QUESTIONS
FROM SFIP FOCUS GROUPS
Part 1: General Environmental Information
Internal Use of Information:
1.
Let's talk about what kinds of general environmental information you use in your work,
and where you go to find it. What types of environmental information are you most
interested in?
For instance, do you use facility level reports, sector level comparisons, performance
data, compliance information, regulatory violations?
2.
For Industry: /fpossible,please addressfacUity and corporate level use for each question.
3.
o
Who (what job functions) uses environmental info at your organization?
EHS, facility managers, etc.
4.
o
o
o
o
o
o
o
What kinds of environmental information does your organization use?
compliance/record keeping
pollutant releases
toxicity of materials
corporate information (how is this environmental?)
demographics? of surrounding area
information on similar facilities
surrounding conditions (water intake/outflow, air)
5.
o
What do you/your organization use environmental information for?
reporting requirements for regulatory programs (CAA, CW A, RCRA, TRl, others if you
have them-name them)
track performance to ensure compliance
ID potential compliance problems
benchmarking against other facilities/companies
provide info to workers
provide info to the public
provide info to community
other Corporate uses
o
o
o
o
o
o
o
6.
How do you benchmark environmental information?
-------
7.
o
o
o
o
o
What makes for good environmental information in general? How is good environmental
information useful?
reproducible
accurate
simple
available to anyone who needs it
what else?
Interaction with the Public:
8.
o
o
o
o
When you deal with the "public," what do you mean by that term? Who is your public?
community next door
media
national groups? Which?
others
Industry: Two forms of interaction: Proactive and reactive to what others do. Let's talk
about proactive communications first.
9.
How do you interact with the public around environmental information? For industry:
What kind of performance info do you give public? How do you decide what's
appropriate? Who decides? Do you give them whatever they ask for?
Reactive:
10. When you give out information, how do you deal with anything less than perfect
performance?
11.
How are you affected by public perception of your data? How do you respond?
Part 2: Experience with SFIP
General Information Use:
12.
o
o
o
Which types of SFIP information do you use, and for what?
Compliance, efficiency info, pollutant profiles
Data for your facility, your company, your competitors, your sector, other sectors?
Detailed facility reports, facility-level stats, sector aggregate stats
-------
13.
o
Is SFIP information "good" infonnation, as you defined it earlier?
reproducible, accurate, etc.
Compliance, Performance, Behavior:
For EPA and NGOs:
14.
15.
16.
Do you think the perfonnance/compliance activities of facilities are likely to be affected
in any way by the existence of the SFIP website? Why and how? changes in operations?
Regulatory attention? Relations with community?
Has SFIP changed anything about the nature of your environmental work? For example,
is there any difference in tracking your submissions to regulators? Is there a difference in
how you go about your internal compliance-assurance activities?
Is there anything different in how you interact with others? In what ways?
Awareness:
17.
18.
Who do you think is interested in the SFIP? Community? State? Regulators? Others?
Have you heard anything from these constituencies about the SFIP? If not, why do you
think this?
How/when did you first become aware of SFIP? Trade association? Regulator?
Regional/state/fed EP A? FR notice? Someone in your organization? Media? Others?
For Industry:
19.
20.
21.
Has public perception of your facility/company been affected by SFIP? In what ways?
Who have you heard from and how? Was it friendly? Have there been articles in local
papers? How have you responded? Do you think the public understands the SFIP data?
Do you think that SFIP changes the public's understanding of environmental infonnation
related to the industries included in the pilot project? How?
Can you suggest ways to improve awareness of SFIP and other information projects?
What would be the best ways to let everyone know about these types of projects?
-------
Future Directions:
22.
o
23.
o
o
o
o
o
o
o
o
Given EP A's commitment to making information available and accessible to the public,
how do you think it could be done better? Put yourself in the driver's seat-how would
you do this if you had to?
examples of how you think this has been done well, either by your organization or by
another?
Is there anything about SFIP you haven't mentioned that you want to say?
anything you especially like
comment process and options (Hotline, website comment page)? Does it work for you?
anything you don't like or would like to see changed?
overall accuracy of data? EP A has done 3 data refreshes so far-is that working?
Is the refresh schedule frequent enough?
any ideas to expand site given currently available data?
format
accessibility of infonnation
-------
APPENDIX 3
SFIP PRESS RELEASE, MAY 1, 1998
-------
&EPA
Unlt8d states
Envtronmental protllldlon
Agency
Environmental News
COmmunications, EdU08t1on,
And Publk; Affairs
(1703)
FOR RELEASE: MAY 1, 1998
EPA EXPANDS COMMUNITY RlGBT.'I'O.KNOW EFFORTS TO INCLUDE
ENVIRONMENTAL PERFORMANCE OF FACILITIES IN FIVE INDUSTRY
SEcrORS
TaDya MeeJda. %02-260-1387
~ part of the Clinton AdminiStration's efrorts to expand the public's right to know, EPA
today announced the Sector Facility Indexing Project, a pilot program that will provide for the
first time comprehensive information through the Internet on the environmental performance of
hundreds offaci1ities in five major industries These data were subject to a thorough quality
review to assure accuracy that included extensive comments by industries and states, the primary
source of this information.
"Today's action is part of the Clinton Administration's commitment to expand the pubUc's
right to know," said EPA Administrator Carol M Browner "Putting high.-qua1ity environmental
infonnation into the hands of citizens is one of the most powerful tool8 for protecting public
health and the environment in our communities The information in this pilot program comes
largely from industrial and state sources and has been subject to a rigorous quality-assurance
process "
The indusuial secton covered include automobile assembly, pulp manufacturing.
petroleum refining, iron and steel production, and the primary smelting and refining of aluminum,
copper, lead and zinc (nonferrous metals)
The ~ covers 653 facilities within the five sectors, and for the first time
coUects in one place infonnation that the facilities must provide under a number of federal
environmental statutes The data include information on past inspections and enforcement actions,
the size of the facilities and their annual releases of chemicals into the envirorunent Demographic
data about communities near the facilities are also included
The database has multiple uses: Facilities can benchmark their data against those of other
similar facilities, or simply monitor their own regulatory performance Environmental and
conununity groups now have easier ae<:esS to information that they can use to learn about the
R-55
-more-
------- |