United States
Environmental Protection
Agency
Office of Environmental
Information
260R-00-001
Office of Information
Analysis and Access
November 17, 2000
LESSONS LEARNED
about designing, developing, and disseminating
Environmental
Information
Products
expanding access and
enhancing the usability of
environmental information
-------
About Designing,
Developing, Disseminating
Environmental Information Products
U.S. Environmental Protection Agency
Office of Environmental Information
Environmental Analysis Division
-------
Abstract
The Environmental Protection Agency (EPA) is committed to expanding access
and enhancing the usability of environmental information under Goal 7 of the
Agency's Strategic Plan. One way to expand access to environmental informa-
tion is for the Agency to enhance existing data and to develop integrated infor-
mation products derived from data compiled to support existing environmental
management and regulatory programs. This is sometimes referred to as the
secondary use of environmental data. The Office of Environmental Information
(OEI) recently conducted a series of interviews with senior information manag-
ers to determine and articulate best practices for designing, developing, and
disseminating information products, especially those based upon data collected
for other purposes.
-------
EXECUTIVE SUMMARY 1
1. INTRODUCTION 5
1.1. Background 5
1.2. Approach 6
1.3. Organization of this Document 7
2. KEY FINDINGS 9
2.1. Design 9
2.2. Development and Review 16
2.3. Dissemination, Maintenance, and Feedback 17
3. IMPLICATIONS FOR OEI 21
APPENDIX A - Information Products Discussed 24
APPENDIX B -Advisory Committee Members 26
APPENDIX C- Interview Questions 27
Exhibit 1. Secondary Use of Environmental Data:
How Data Become Information Products 8
Exhibit 2. The Six Principle Aspects of the
Information Product Life-Cycle 9
Exhibit 3. Stakeholders, Customers, and the Public 10
Exhibit 4. Phases of Stakeholder Input 12
Exhibits. Illustrative Information Partnership 15
Exhibit 6. Affected Parties by Product Types 18
Exhibit 7. Elements of a Strategic Information Product
Planning Process 23
November 17, 2000
-------
-------
EXECUTIVE SUMMARY
The Environmental Protection Agency is committed to expanding access and
enhancing the usability of environmental information under Goal 7 of the
Agency's Strategic Plan. One way to expand access to environmental information
is for the Agency to enhance existing data and develop integrated information
products based on data inventories that support environmental management and
regulatory programs. It is important to evaluate information products as part of an
overall effort to assure customer satisfaction and continuous improvement.
This Lessons Learned report focuses entirely on one environmental information
gateway to the public: government information products. Environmental informa-
tion products produced by EPA and states vary greatly in terms of management
and implementation. As a result of their varying design and development proto-
cols, the Office of Environmental Information sought to examine past efforts to
develop information products in order to delineate lessons learned that may be
helpful to future endeavors. Information contained in this report was derived
primarily from interviews conducted with agency staff involved with the develop-
ment and use of a variety of information products in order to understand and build
upon their experience developing and managing information products. The
ultimate goal of this project was to determine if there were some common lessons
learned across various types of products and product developers.
An effort was made to incorporate a range of different product types. These
include those driven by statute or regulation (e.g., the Air Trends and TRI re-
ports); compliance-based products (e.g., Pennsylvania Compliance Reporting
System and the Sector Facilities Indexing Project); those designed to enhance
data management and compliance (e.g., Region 3's Enviroviz, and the Office of
Water's Index of Watershed Indicators); and those involving numerous external
stakeholders (e.g., the South Baltimore Pilot Project).1
As noted above, we focused on identifying overarching issues rather than
limiting the investigation to any one information product or category of prod-
ucts. However, our analysis did recognize that the lessons learned will vary
according to specific product type and category. For example, compliance
products can be controversial due to the response of specific facilities and
1 Please see appendices for a complete list of the products evaluated, brief descriptions of each
project, and a list of the individuals interviewed.
November 17, 2000 Page 1
-------
individuals to Agency programs. In this report, we have identified a set of
issues and concerns that are generally applicable to the design and management
of EPA and state information products. Where appropriate, we have referenced
specific information products when discussing lessons learned.
Key lessons drawn from the interview process include the following:
Almost all interviewees agree that it is critical to articulate a clear sense
of the product's purpose early in the design process. As obvious as this
may seem, interviewees stress that this step is frequently overlooked,
resulting in redundancy among products, suboptimal user functionality,
and other problems.
It is a mistake to develop a generic information product. While the reality
that information products circulate beyond their intended audience
should be considered during planning, it should not be used as a rationale
for avoiding the need to specify a target audience and its unique needs for
environmental information.
Stakeholder and product audience input must be sought at all points in
the information product life-cycle. Early stakeholder involvement is
especially important as it can help clarify goals, reduce skepticism, build
consensus, and ensure the dissemination of information that meets
audience needs.
Data ownership is directly related to data quality concerns. While some
users of secondary information products take the initiative and communi-
cate effectively with originators of data, EPA lacks an organized frame-
work to guide this kind of technical interaction. Many interviewees stress
the importance of developing a mechanism to facilitate communication
among producers and users of EPA data.
To address concerns over data accuracy, some developers and managers
interviewed note the importance of a "data feedback loop," through
which facilities and other data providers are requested to verify data used
in a particular information product. Beyond sending the data back to
providers, some interviewees feel that the Agency should also consider
initiating an internal comment period via an intranet-accessible form,
allowing adequate time to make corrections prior to public release.
It is necessary to consider all aspects of information usability. Misunder-
standing and misapplication of information products can be avoided by
providing users with comprehensive and carefully articulated metadata
and contextual information.
Page 2 November 17, 2000
-------
As a result of these findings, the OEI, Office of Information Analysis and
Access (OIAA) has embarked on developing a Best Practices series for infor-
mation product development. Building on the feedback and recommendations
from this report, OIAA will lead the Agency effort to develop and release
guidance documents on each important phase of an information product's life-
cycle. Each guide will consist of EPA examples of best practices, case study
applications of these practices, and reference pertinent policy and guidance
already in existence for each topic area. The overarching objective of the series
is to disseminate helpful practices to EPA information product developers and
managers. In doing so, OIAA and partners strive to heighten Agency informa-
tion sharing and best practice implementation for environmental information
products, thereby enhancing the credibility, usability and defensibility of EPA
products.
November 17, 2000 Page 3
-------
What is an Information Product?
A joint EPA/State Action Team was recently convened to design
and publish a periodic bulletin to inform stakeholders and the
public of development and release of significant information
products. The joint team defines a significant information product
as: "A product under development or modification by EPA which
derives from federal, state, local, tribal, or other organizations'
data, and a state product that is regional or national in scope and
aggregates data from more than one state. Such products often
generate considerable attention when they use [pre-existing] data
to describe environmental conditions, trends, potential risks, and/
or portray compliance or performance."
Significant information products include the following:
Products that analyze and/or compare data from various
agencies and organizations, including industry, as well as
various federal, state, tribal and local agencies;
Significant raw unanalyzed data collected by EPA from
various agencies and organizations;
Products that describe or assess environmental conditions,
trends, or risks;
Products that apply to a large segment of the population or
large geographic area;
Models used by the public to perform environmental analyses
based upon data from various agencies and organizations; and
Annual reports and other products released on a regular basis
that describe environmental conditions, trends, risks, and/or
portray compliance or performance.
Page 4
November 17, 2000
-------
1. INTRODUCTION
1.1. Background
Information about the environment - environmental characteristics; physical,
chemical, and biological processes; and chemical and other pollutants - under-
lies all environmental management decisions. Access to quality information and
analytical tools is essential for public decision making, measuring environmen-
tal improvement, assessing and managing risks, and evaluating progress. Ac-
cess to high-quality, user-focused information products will enable the public to
understand environmental conditions and make informed decisions about their
communities.
The Environmental Protection Agency articulates its commitment to expanding
and enhancing access to environmental information in its Strategic Plan. Goal 7,
Expanding American's Right to Know About Their Environment, stresses the
importance of developing and disseminating information products that will
enable the public, governmental agencies, non-governmental organizations and
private sector parties to easily access relevant, usable, and quality-assured data
and information products.
The vision of the EPA Office of Environmental Information (OEI) is to "ad-
vance the creation and use of data as a strategic resource to enhance public
health and environmental protection, inform decision making, and improve the
public's access to information about environmental conditions and trends." Key
goals of the Office of Environmental Information include the following:
Promote the public's right to know,
Improve information infrastructure,
Reduce burden,
Foster information partnerships,
Improve data quality, and
Foster data integration.
The Agency's approach to information product development and dissemination
is relevant to all of OEI's goals. It is therefore critical to understand what makes
information products effective, widely used, and technically credible. It is
especially important to assure the quality and user applicability of information
products derived from existing data.
It is...critical to
understand
what makes
information
products
effective,
widely used,
and technically
credible.
November 17, 2000
Page 5
-------
To this end, the Environmental Analysis Division (BAD) within the Office of
Information Analysis and Access interviewed senior staff involved with the
development of major environmental information products within and outside
the Agency. The lessons learned from this activity serve as a resource for
formulating information product guidance and will support a variety of efforts
to foster continuous improvement in the development and dissemination of EPA
information products. The themes and implications outlined in this document
are part of an overall effort to characterize best practices and ensure continuous
improvement in EPA information products, services, and policies.
1.2. Approach
As indicated above, the
findings and recommenda-
tions articulated in this report
were derived through a series
of interviews with senior
EPA and state staff involved
in the development and/or
administration of environ-
mental information products.
Working with an Advisory
Committee of senior EPA
staff and managers, OEI
selected interviewees closely
involved with projects result-
ing in a variety of informa-
tion products, including
paper-based reports, web-
based products and information applications, and multi-media products with
geographical scopes of interest ranging from local to national perspectives.
Many of these information applications utilize data collected for other purposes.
All interviewees responded to the same questionnaire, and the research team
focussed on common issues rather than investigation and critique of individual
products. The products evaluated through the interview process can be grouped
in terms of four broad categories:
Information Projects Reviewed
/ Environmental Monitoring for Public
Access and Community Tracking
(EMPACT)
/ Index of Watershed Indicators (IWI)
/ National Air Quality and Emission
Trends Report
/ Pennsylvania Compliance Reporting
System (PA CRS)
/ Enviroviz
/ Sector Facility Indexing Proj ect (SFIP)
/ South Baltimore Pilot Proj ect
/ Toxics Release Inventory (TRI)
Information products that are driven by statutory or regulatory reporting
requirements, such as the annual Toxics Release Inventory (TRI) and Air
Quality Trends reports.
Information products developed or used to facilitate regulatory compli-
ance, such as the Pennsylvania Compliance Reporting System (PA CRS)
and the Sector Facility Indexing Project (SFIP).
Page 6
November 17, 2000
-------
Information products designed to enhance data management or facilitate
the integration of diverse data sets, such as Enviroviz and the Index of
Watershed Indicators (IWI).
Information products that involve extensive stakeholder input and/or
review, such as the South Baltimore Pilot Project.
While diverse in many respects, the information products discussed during the
interviews are similar because they are all based upon numeric, statistical, or
graphical data. Most of these products are developed for secondary uses, a
category that includes most information products from OEI. As illustrated in
Exhibit 1, information products for secondary use are frequently based upon
data extracted from EPA program systems. EPA data systems are used to sup-
port or administer regulatory programs, whereas secondary information prod-
ucts are designed to address other purposes. The information products reviewed
during the interview process are all "data rich," as opposed to those intended
only to disseminate a message, broadcast the occurrence of an event, express an
opinion, or describe an activity or program. Although all public communica-
tions are important and require thoughtful formulation, data-based information
products present a special set of challenges. For instance, data-based informa-
tion products require careful attention to factors such as accuracy, quality and
precision, and the ability to support integration across multiple systems. These
and other attributes of the primary data must be carefully considered in the
design, development, and dissemination of information products intended to
serve secondary purposes.
Although all
public commu-
nications are
important and
require
thoughtful
formulation,
data-based
information
products
present a
special set of
challenges.
1.3. Organization of this Document
The remainder of this document is organized around two principal sections:
(1) a summary of key findings organized in terms of major phases of the infor-
mation product life-cycle: Design; Development and Review; and Dissemina-
tion, Maintenance, and Feedback; and (2) a concluding section that highlights
implications for the Office of Environmental Information and recommendations
for the success of EPA information products.
The document also includes three appendices: (A) Summary of Information
Products Considered, (B) Advisory Committee Members, and (C) Interview
Questions.
November 17, 2000
Page?
-------
Exhibit 1. Secondary Use of Environmental Data:
How Data Become Information Products
Data Providers
Regulated concerns
States
NGOs
Needs Assessment
Decision-Makers
& Information Users
EPA Data Systems
Secondary use
of environmental
data and
development of
information
products
Most EPA data are initially collected or generated by first-order data
providers, usually industry or other regulated entities and states. Data are
then transferred from the original provider to EPA/state data systems,
where they are used to support various regulatory and program administra-
tion activities. The data collected by EPA/state programs can also be used
or adapted for use in secondary applications. Secondary applications
include integration, consolidation, analysis, and context-specific uses of the
original program data. The offices within the Agency, particularly the
Office of Environmental Information (OEI), undertake various information
development and enhancement initiatives, resulting in "secondary informa-
tion products," which are used by a wide range of decision-makers at the
local, state and national level. Use of these information products lead to
environmental and/or right to know outcomes. As suggested by the sche-
matic (above), feedback from information users and decision-makers helps
to improve information products. The process of moving from data to
effective information products is inherently iterative and requires coordina-
tion with data providers and stakeholders.
Page 8
November 17, 2000
-------
Exhibit 2. The Six Principle
Aspects of the Information
Product Life-Cycle
1. Design
2. Development
3. Review
4. Dissemination
5. Maintenance
6. Feedback
2. KEY FINDINGS
As illustrated in Exhibit 2, the
information product life-cycle can
be characterized in terms of six
aspects: design, development,
review, dissemination, mainte-
nance, and feedback. Findings
from the interviews suggest
important overlaps among these
phases. For this reason, findings
have been combined and pre-
sented in three basic groups:
design; development and review;
and dissemination, maintenance
and feedback.
Placement of findings within the life-cycle framework will help to guide future
product developers in a logical and progressive manner. As previewed below,
the lessons derived through this study highlight the importance of a life-cycle
approach to information product management.
Design. The design stage of an information product includes defining the
purpose and audience, aligning product scope and user needs, involving
stakeholders, coordinating with other relevant efforts, defining data
suitability criteria, addressing resource issues, and developing a product
plan.
Development and Review. The development phase requires input from
customers, stakeholders, and data providers; consultation with data
providers to assure suitability and accuracy; and configuration of prod-
ucts that are consistent with metadata.
Dissemination, Maintenance, and Feedback. The dissemination stage
goes beyond merely providing access to the product to ensuring timeli-
ness, providing metadata, providing contextual information, and seeking
and utilizing continuous feedback from users and stakeholders.
2.1. Design
The majority of interviewees strongly recommend adoption of a strategic
approach in the information product design phase. Accordingly, information
project managers should carefully consider the goal of the project, articulate
intended audience and user needs, determine criteria to ascertain whether data
collected for one purpose are suitable for other purposes, and identify possible
November 17, 2000
Page 9
-------
linkages with Agency partners with whom to collaborate on product develop-
ment. Interviewees explain that early strategic planning allows project managers
to better anticipate and respond to potential changes in a project's budget, goals,
and target audience.
Identify Product's Purpose and Audience
Almost all interviewees agree that it is critical to articulate a clear sense of
the product's purpose early in the design process. As obvious as this may
seem, interviewees stress that this step is frequently overlooked, resulting in
redundancy among products, suboptimal user functionality, and other prob-
lems. As part of this process, it is important to identify primary audiences for
the product, key stakeholders, and potential sponsors within and beyond the
Agency (see Exhibit 3). Clear understanding of the purpose and the audience
will help determine the methods used to compile and present the information
for dissemination. Consequently, front-end goal articulation and audience
identification are critical to the success and usability of an information
product.
Align User Needs and Product Scope and Character
Most interviewees recognize the importance of delivering a product that
meets the specific needs of an intended audience. Customer needs vary
depending on the users' levels of proficiency in data use or analysis and their
ultimate plans for the information. Interviewees caution that the actual
Exhibit 3. Stakeholders, Customers, and the Public
General Public
All of EPA's information products are
developed and disseminated to serve
the public's right to know about
environmental conditions and trends.
However, it is helpful to distinguish
between "general public," "stake-
holders," and "customers" for an
information product. Customers are
individuals or groups who will use
the product. Information products should be developed to address the
needs and capabilities of customers. Stakeholders are individuals or
groups who will likely be affected by the use of the product. Stakeholders
are frequently the original providers of data used in developing an infor-
mation product. Although it is important to distinguish between stake-
holders and customers, it is also important to recognize that stakeholders
and customers sometimes overlap.
Page 10
November 17, 2000
-------
audience frequently extends beyond the intended audience. This requires
managers to consider all potential users when defining the scope of a project.
For instance, IWI was originally targeted at states as a tool to spark place-
based discussion of water quality issues. Project managers recognized its
potential value to the public, and expanded the scope of the project accord-
ingly. While the reality that information products circulate beyond their
intended audience should be considered during planning, it should not be
used as a rationale for avoiding the need to specify a target audience and its
unique needs for environmental information.
Interviewees emphasize that product developers should take care to distin-
guish between a user's need for raw data, interpreted results, or a summary
report. They stress that product managers should adjust the degree of data
interpretation according to the level of expertise of the expected audience.
Interviewees also note that information products should avoid certain types
of interpretation. Specifically, the Agency has often run into difficulty when
attempting to rank facilities or areas based on available data. Interviewees
stress that information products should not rank aspects of their information
(e.g., facility compliance, relative stream quality, air quality variance, etc.).
Instead, they should provide information that enables users to conduct such
analyses on their own.
During the interviews, information product managers and designers reported
on ways they address the issue of data interpretation when required by their
products. Examples include:
/ Informing stakeholders of the expected level of interpretation in the
early phases of a project (PA CRS);
/ Enabling users to adjust the algorithms that calculate ranking, risk, or
similar assessments (IWI); and
/ Targeting projects to local communities interested in specific sets of
interpreted data (EMPACT).
Early Stakeholder Identification and Involvement
In addition to identifying and involving a product's audience, interviewees
indicate that product teams should identify essential stakeholders and involve
them early in the design phase. This distinction between users and data provid-
ers will vary and even overlap depending on the product, but interviewees felt
it was essential for product developers to consider the needs of both the exter-
nal user and the identified stakeholder group(s). This frequent overlap can be
easily understood if adequate time is devoted to identifying the various parties
(see Exhibit 3). Oftentimes, product developers consult only stakeholder
groups, or "user groups", as a means of soliciting feedback. Some interviewees
November 17, 2000 Page 11
-------
suggest that such limited external input is not only inadequate, but that feed-
back is often misleading and perhaps even related to the perceived failure of an
information product by the user community.
Early stakeholder involvement is especially important as it can help clarify
goals, reduce skepticism, build consensus, and ensure the dissemination of
information that meets audience needs. Exhibit 4 demonstrates how the scope
and nature of stakeholder input can vary across the product life-cycle.
Interviewees stress that while the initial cost of engaging stakeholders may
exceed perceived benefits, in the end actual benefits will be higher than the
initial cost. Resources may be utilized for product development and implemen-
tation rather than to justify and defend the information product to stakeholders.
The PA CRS provides a good example of the importance of involving stake-
holders, especially when the product is perceived as controversial and faces
criticism or resistance. PA CRS managers met frequently with industry
representatives clearly indicating their intentions regarding the final product
while being receptive to industry's concerns. Despite initial doubts, the
affected industries participated in the design of the final product.
While interviewees advocate early involvement of stakeholders, they also
believe there are limits to helpful stakeholder input. One respondent indi-
cates that Agency experts need to spend more time determining what infor-
mation they want to collect and why, before inviting stakeholders into the
process. He believes that getting divergent opinions before having a clear
Exhibit 4. Phases of Stakeholder Input:
Diameter of circles suggests comparative degree of input
1. 2. 3.
Preliminary input Canvassing stakeholders, Broad-scale public
reanalysis if necessary input and comment
4.
Product development;
Evaluate input and
modify product scope
Specifically - Stakeholder
impacted decision samples
makers - Decision makers
Specialists - Analysts
Stakeholder
representatives
Public (if
appropriate)
Decision makers
5.
Reaction to product
- Specifically - Specifically
impacted decision impacted
makers stakeholders and
- Specialists decision makers
- Specialists
Page 12
November 17, 2000
-------
strategic plan results in confusion. The majority of interviewees agree that
asking structured questions about a product results in the most manageable
and useful form of feedback.
Determine Criteria to Ascertain Data Suitability
As discussed previously, many environmental information products tend to be
data rich. It is therefore important to ensure that data utilized for these applica-
tions are suitable and appropriate for the anticipated use. The suitability of data
may be determined by addressing the following questions:
What does the database cover? It is important to identify the kinds of
information contained in the database (e.g. types of pollutants, facilities,
permits, and acquisition methods).
Can the database be used for spatial analysis? Product developers should
determine whether the data are available at appropriate geographical scales
such as ZIP code, latitude and longitude, county code, state code, etc.
Can the database be used for temporal analysis? Product developers should
determine whether the data are collected on a fixed schedule (daily,
monthly, yearly, etc.) and if the data are sufficiently consistent overtime to
allow period-to-period comparisons.
How consistent are the variables over space and time? It is crucial to assess
the degree of internal consistency, allowing comparisons across space
(facilities, Regions, etc.) and over time (monthly, yearly, etc.)
Can data be linked with information from other databases? Product devel-
opers should determine the degree to which information can be linked with
other databases based on common characteristics such as facility identifica-
tion numbers, latitude and longitude, geographical codes, etc.
How accurate are the data? Product developers should seek and consult
information from data quality checks performed by the Program Office as
well as from statistical analysis performed by other organizations.
What are the limitations? Each of EPA s databases has a primary purpose
for which it was developed and is maintained. As the databases are as-
sessed for suitability for alternate uses, it is important to understand the
constraints and limitations of the database.
How can I get information? It is crucial to identify formats in which the
database is available such as printed form, diskettes, CD-ROM, online
access, etc., along with names, addresses, and phone numbers to contact for
detailed information.
November 17, 2000 Page 13
-------
Is there documentation? A quality database requires documentation to
support it, such as information on data collection methods, quality assur-
ance mechanisms, data management, users' guides, and information dis-
semination. Details on the availability of such documentation should be
provided.
Coordinate with Other Relevant Efforts
Interviewees feel it is important to ensure that information product develop-
ment is coordinated among similar efforts underway elsewhere at EPA, other
federal agencies, or states. Adequate coordination will limit duplication of
efforts, increase the utility of EPA data, and help ensure that Agency goals
are being addressed. Several interviewees suggest that establishing Agency-
wide procedures will result in a stronger relationship between information
products and EPA's goals, providing managers with a meaningful context
from which to measure product performance.
One interviewee emphasizes the importance of identifying external partners
who are committed to the project and have the expertise to provide construc-
tive assistance, especially in the early design phases of an information
product. Effective partnerships between similar efforts can also help overcome
budget constraints, promote data sharing, and increase data quality. Exhibit 5
portrays an illustrative partnership between EPA and a regulated industry.
EPA's Environmental Monitoring for Public Access and Community Track-
ing (EMPACT) Program, for example, provides funding to initiate commu-
nity-level projects and encourages local partners to assume financial
responsibility after EPA funds are exhausted. Data quality is enhanced
through reviewing methods and sharing data by experts involved in similar
efforts. Sharing information and coordinating efforts should go beyond the
Agency to other federal agencies, as developers of IWI indicate.
Address Resource Issues
Resource considerations play a significant role in determining project suc-
cess. This fact is especially true for information products with relatively less
up-front Agency funding or support. Products mandated by legislation or
regulation are often better funded than products that stem from user feed-
back, internal collaborations, or community-based partnerships. While a
clearly defined budget should be developed in the early stages of the product
life-cycle, it is necessary to maintain flexibility and revise the budget on
encountering unexpected contingencies. Perhaps most importantly,
interviewees stress that information product development requires a long-term
commitment, suitable to support a product as it evolves through the design,
development, stakeholder review, and dissemination and feedback phases.
Page 14 November 17, 2000
-------
Exhibit 5. Illustrative Information Partnership
Industry
In some cases, EPA obtains data from regulated industries, often through
other government agencies (frequently at the state level). Under such a
configuration, EPA and its state and/or industry partners can collaborate on
efforts to improve public access and usability.
Budget planning should clarify both resource needs and availability. The
definition of resource needs should extend beyond the early stages of product
development and include all aspects of product design, development, and
dissemination and feedback. While specifying resource availability, product
planners should earmark initially available funds from the Agency as well as
identify potential funding sources, internal and external to the Agency, to
ensure successful completion of the project. As an example, developers of
the South Baltimore Pilot Project note that they established their goals based
on the size of the initial budget in order not to take on too much at one time.
Even so, the project exceeded its budget and funds have not been made
available to publish all findings.
Develop a Product Plan
Several product developers interviewed for this study emphasized the need
for a product plan, which would formally contain strategic planning consid-
erations relating to information product development and management. The
product plan would contain three key elements:
A summary of current circumstances, including explicit user-focused
rationale for the product;
A schedule and process description dealing with how the project team
will approach each aspect of the product life-cycle (see Exhibit 2); and
A clear statement of the outcomes expected to result from the product.
The plan should also address key tactical issues associated with the use of
existing data, such as data appropriateness and accuracy, methodological
issues, peer review procedures, data display and access issues, and general
management of product content.
November 17, 2000
Page 15
-------
2.2. Development and Review
Traditionally, resources and activities tend to be concentrated on activities associ-
ated with the actual development of an information product. The focus is on
methodological issues, testing, and review of the product. Interviewees express
concern about this limited field of activity and articulate several issues requiring
increased attention during the development phase of the product life-cycle.
Obtain Stakeholder Input During Product Development
As with product design, interviewees stress the importance of stakeholder
input during product development. However, participants are divided on
whether to seek open-ended or structured stakeholder input. Some
interviewees indicate a preference for soliciting feedback on developed
concepts rather than utilizing an entirely open-ended, unstructured approach.
Representatives from TRI and the SFIP note the importance of thoroughly
developing concepts and product drafts prior to soliciting stakeholder feed-
back, arguing that such an approach leads to more constructive feedback.
Other interviewees contend that stakeholder input should be gathered to
affect the product revision and redesign that tend to occur during develop-
ment of product components. In one example, IWI managers worked with
states and other stakeholders to complete a review of the overall program,
the rankings, and the underlying methodology - a process that resulted in
valuable contributions to the final product.
Recognize the Relationship between Data Acquisition, Use, and Quality
Data ownership is directly related to data quality concerns. While some users
of secondary information products take the initiative and communicate
effectively with originators of data, EPA lacks an organized framework to
guide this kind of technical interaction. Many interviewees stress the impor-
tance of developing a mechanism to facilitate communication among produc-
ers and users of EPA data. This is especially important when data are being
used for purposes other than those of their original collection. Respondents
also express a desire for Agency-wide guidelines on conducting quality
control protocols on data prior to using it for a secondary purpose.
An important question for interviewees is, "who should be responsible for
ensuring data quality?" In many cases the concept of data quality is used
casually by information product developers and designers as a way to refer to
data accuracy and error correction procedures. This interpretation provides
only a partial understanding of data quality, and does not sufficiently empha-
size the fact that quality is a function of intended use. What one user considers
data of high quality, another might deem wholly inadequate. A user's perspec-
tive on data quality often extends beyond data accuracy into other data charac-
teristics. For instance, is the data generalizable (for example, was it collected
Page 16 November 17, 2000
-------
using a probabilistic sampling model? Is this method for collection similar to
other data sets?), is the data current (annual, monthly or periodic updates), or
are there reproducible results (how thorough is the metadata and documenta-
tion information)? This is particularly relevant in cases where users must work
with data they did not collect. In these situations, it can be presumed that users
are not aware of the purpose behind the original data collection. Interviewees
emphasize that information product developers must obtain and examine all
available documentation, guidelines, and contextual information to assure that
data are suitable for use in new, possibly unintended applications.
Improve Data Accuracy with Input from Data Providers
To address concerns over data accuracy, developers and managers of SFIP,
IWI, and the South Baltimore Pilot Project note the importance of a "data
feedback loop," through which facilities and other data providers are re-
quested to verify data used in a particular information product. In addition to
assuring accuracy, such a procedure helps address concerns that information
providers might have regarding the projects. Beyond sending the data back to
providers, interviewees feel that the Agency should also consider initiating
internal review procedures or a comment period via an intranet-accessible
form, allowing data to mature internally prior to public release. For example,
one respondent explained that his managers wanted an assessment of the
quality of the data used in Enviroviz. Lacking established Agency standards,
he developed a 3-level set of standards that he could then apply to data used
in Enviroviz.
Continuously Involve and Utilize Feedback from the User and Informa-
tion Provider
Feedback from product users and information providers is essential to de-
velop useful products and improve data quality. Most interviewees stress the
importance of eliciting feedback about data quality, usability, and clarity
from product users and information providers. As illustrated in Exhibit 6,
interviewees recognize that the need for input from users and data providers
will vary depending on product type and affected parties.
2.3. Dissemination, Maintenance, and Feedback
The ultimate success of an information product is reflected by the reaction of
its audience. Interviewees emphasized issues relating to user satisfaction
such as timely release, data quality, metadata, contextual information, and
continued solicitation of feedback.
Remain Committed to Information Availability in a Timely Manner
Interviewees stress the importance of making environmental information
available in a timely manner. When the research team began this project, a
November 17, 2000 Page 17
-------
Exhibit 6. Affected Parties
Affected Parties Role
by Product Types
Sample Product Types
Regulated community
Users
Regulated community, Data
state and local governments providers
Regulated community, Data
state and local governments, providers
interested members of the and
general public users
Models to estimate emissions,
calculate compliance costs, and
identify substitute inputs
Software for inputting data or
transferring data to EPA
Status and trends tool
primary concern was the degree to which Agency data managers might be
hesitant to release information for fear of its inappropriate use. For example,
some non-government organizations may use data from TRI or the Air Trends
Report and include it in area rankings in a manner that EPA would not support.
The research team raised this issue with each interviewee. In almost all
cases, they indicate that they recognize the problem, but view it as beyond
the Agency's control. Some respondents suggest that managers should make
an effort to anticipate secondary uses and do their best to consider them as
they complete product development, while recognizing the impossibility of
foreseeing all possible uses for Agency data. Above all, interviewees said
that the Agency cannot let this concern interfere with the timely dissemina-
tion of information.
Rather, the Agency should work to ensure that products are derived from
suitable data and are packaged with sufficient metadata and other descriptors
of appropriate use to minimize the misuse of the primary data. All data-based
information products should include relevant metadata (i.e., data about the
data) and clearly identify the intended use of the product to minimize, if not
eliminate, possible misuse of data. Agreed-upon data quality and suitability
goals can be integrated into a product's design and facilitate timely release.
Approach Dissemination as an Opportunity to Improve Data Quality
Some interviewees argue that data quality will improve through widespread
dissemination and use. This group argues that users will offer feedback, and
in turn, Agency commitments to the public's right to know will not be
compromised with lengthy internal reviews. The impetus for this view is
based on the following logic: improved access and display will lead to
Page 18
November 17, 2000
-------
increased use, increased use leads to increased scrutiny, increased scrutiny
will identify discrepancies, and the Agency will respond by correcting errors
and/or clarifying misperceptions.
Provide Customers with Easy-to-Understand Metadata
The term "metadata" refers to information about a data-based product's
content, technical configuration, and the methods used for analysis and/or
data collection. Metadata is needed so that users can assess data quality and
appropriate use in relation to their own needs. The development and place-
ment of adequate metadata within information products is a particularly
important issue for EPA because secondary users constitute a significant
audience for the Agency's information products. While EPA is making
widespread public access to environmental information a reality, it is critical
that all information products provide sufficient documentation for their
intended audience.
Furthermore, by defining data suitability objectives and making them avail-
able as part of the metadata, product designers can avoid potential misunder-
standings and clarify the known quality of the data. For example, SFIP
managers received several complaints about displaying out-of-date data;
however, since SFIP is only updated once every quarter to half-year, users
should expect some time lags. Therefore, it is important to indicate how
often the data are refreshed or updated using simple, non-technical language.
Provide Sufficient Context when Releasing Data
Beyond the provision of metadata, several interviewees suggest that, where
appropriate, products need to include an overarching statement of context as
another way to orient users and avoid potential misinterpretation. For ex-
ample, several interviewees express concern that releasing monitoring data
may lead to the wrong impression that states with more comprehensive
monitoring or enforcement programs have more serious environmental
problems than other states that collect and/or maintain comparatively less
data. Interviewees suggest that a solution might be to include information
describing the extent and nature of a state's monitoring network(s). This
concern extends beyond a previous finding which calls for adequate
metadata with information products.
A similar problem arises whenever new or enhanced monitoring is imple-
mented or displayed. As one interviewee explains, methodological changes
for estimating emission trends in the Air Trends Report resulted in data and
graphs that did not look the same as they had in the past. This caused some
readers to mistakenly infer changes in environmental outcomes. He sug-
gested that increased attention to contextual information would help to avoid
such misinterpretation.
Improved
access and
displaywill
lead to in-
creased use,
increased use
leads to
increased
scrutiny,
increased
scrutiny will
identify dis-
crepancies,
and the
Agency will
respond by
correcting
errors and/or
clarifying
misperceptions.
November 17, 2000
Page 19
-------
Seek and Facilitate Feedback and Continued Communication with Data
Users and Providers
Communication should extend past the design and developmental phases of
an information product, and continue after the product's release. Maintaining
communication with data providers will enable product designers to solicit
and receive corrections and updates. Several interviewees note that the
realization that information will be analyzed and distributed is a motivating
force behind efforts to improve data quality. For example, IWI currently
includes a designation noting cases when insufficient data exist to draw a
conclusion about water quality. When states or other users see the designa-
tion applied to their information, they often seek to increase reporting and
provide additional data, therefore improving data quality.
Page 20 November 17, 2000
-------
3. IMPLICATIONS FOR OEI
The EPA Office of Environmental Information acts as the steward of the
Agency's environmental information, ensuring that EPA and other concerned
organizations collect, develop, and disseminate high-quality and useful infor-
mation about environmental conditions. Although focused on serving the
public's right to know about environmental issues, OEI aims to move beyond
the provision of existing data and generate new information products that are
responsive to identified user needs. Increasingly, environmental information
products are derived from data originally collected to support state or federal
regulatory programs or management activities. Assuring the suitability of these
data for new applications is therefore of paramount importance.
After completing the interviews with information product designers and manag-
ers, the research team focused on identifying a set of overarching lessons
learned and their implications for the Office of Environmental Information.
Summarized below, these lessons will serve as the basis for a series of "Best
Practices" reports. The best practices will help EPA product developers and
managers assure that information products are useful, credible, and fully appli-
cable to user needs and access preferences. Organized around the themes
outlined below, the best practices series will compile existing knowledge and
documentation, and when necessary, introduce new methods and approaches to
assure that EPA information products are always "on the mark" with respect to
user needs and environmental decision making.
Avoid Defining Your Audience as "The Public"
EPA is currently striving to expand and enhance its role as a communicator
of environmental information. OEI should develop guidelines to help ana-
lysts integrate product performance goals with the needs of specific target
audiences. Such guidelines would help the Agency anticipate how the utility
of a particular product might vary from group-to-group. Depending on the
audience, developers can incorporate various types of software formats,
technical language, embedded caveats, collection context, and helpful hints.
Beyond the presentation of information, the rate of error correction or prod-
uct updates could also depend on the intended use and corresponding audi-
ence. Issues such as these are easy to overlook when the external audience is
classified as "the general public."
Data Quality is a Function of Data Use
Data quality is not a generic concept. It is critical that data quality be consid-
ered in light of its intended use. The quality of data is a function of its char-
acteristics and limitations in the context of a particular application. OEI
should develop guidelines and resources to help product planners and
Agency analysts judge the suitability of data for particular user applications.
Without best practices concerning the suitability of existing data for antici-
pated applications, the credibility of EPA information products may suffer.
November 17, 2000 Page 21
-------
Assure Credibility with Methods Review and Documentation
A standardized approach to assure that information products are planned and
developed using appropriate analytical tools, documentation, and review
procedures will help to increase the credibility and usability of information
products. Although all EPA products, prior to dissemination, are subject to
review by the originating office's designated communications officer, OEI
should develop Agency-wide guidelines for methods review and documenta-
tion of the development of an information product. The latter is especially
important as it assures replicability of results.
Develop Guidelines for Stakeholder Involvement
Depending on the nature of the information product and its intended use, the
appropriate type, scope, and timing of stakeholder participation can vary
significantly. Input can range from comments on both the products themselves
and/or the raw data contained within a product. OEI could help develop guide-
lines for stakeholder involvement, focussing on how input might vary with
level of product maturity, phase of the product life-cycle, and other key factors.
Information Customers Need Metadata
OEI has a lead responsibility in developing Agency-wide guidelines for
metadata to be included with information products. Increased attention to
preparing metadata should be a key component of the Agency's public access
efforts as it is critical to ensuring the development of useful and credible
information products and assists secondary uses of Agency information.
Facilitate Life-Cycle Information Product Planning
The factors discussed above must be approached in an integrated manner. As
a final step, OEI should develop comprehensive information product plan-
ning procedures to assist program offices in evaluating the long-term goals
for designing and maintaining information products. The guidelines should
help managers consider target audiences, appropriate stakeholder involve-
ment, quality assurance procedures, budget requirements, and other key
product development and maintenance factors. Resources developed by OEI
should be formulated and couched to help planners address the full life-cycle
of an information product (see Exhibit 2).
The Upshot: A Need for "Best Practices"
The Environmental Protection Agency strives to gather, utilize, and provide
reliable and timely data to make decisions, improve information management,
document and measure performance, reduce reporting burdens, and enhance
public access. OEI's OIAA sought to reexamine past information product
efforts in order to delineate lessons learned that may be helpful to future en-
deavors of information product design, development, and dissemination as they
feed into these overarching themes. The interview process did not result in any
Page 22 November 17, 2000
-------
"magic bullets" to assure perfect information products. Interviewees did, how-
ever, recommend a variety of practices that tend to contribute to the effective
design, development, and dissemination of information products. The individu-
als interviewed for this project share decades of experience working with
environmental information products. They shared both successes and failures,
what to do as well as what to avoid. Overall, the findings demonstrate that EPA
has a wealth of information product experience that is not being adequately
communicated to information product developers who could learn from it.
However, beyond information sharing and technology transfer, a perceived gap
exists for guidance on sound analytical approaches to the reoccurring data
concerns, specifically for information product applications.
As a result, OEI will now embark on the Best Practices Series, a series of Agency
guides on information product development. The purpose of this series is to help
information product developers create products of known quality and usability for
an external audience. In doing so, OEI and partners will strive to heighten Agency
information-sharing and technology transfer and reduce internal burden, thereby
enhancing the credibility, usability and availability of EPA products and ulti-
mately providing more information of higher quality to the user. Analysis of the
information captured through this interview process suggests that best practices
tend to reinforce one another. Not only is it important to articulate and heed best
practices, it is important to do so in an integrated manner. Depicted in Exhibit 7,
below, adoption of an integrated approach to information product management
would help to ensure continuous improvement, enhance the public's right to
know, and support the mission of the Office of Environmental Information.
The best practices series
will also help EPA and state
program managers as they
oversee regulatory and
administrative data systems,
helping them to anticipate
new uses for their data. In a
similar vein, the best
practices will be useful to
grantees and other external
stakeholders who collect
environmental data, assur-
ing the ever increasing
suitability of data for
unforeseen applications and
the overall usability of EPA
information.
Exhibit 7. Elements of a Strategic
Information Product Planning Process
Not only is it
important to
articulate and
heed best
practices, it is
important to
do so in an
integrated
manner.
November 17, 2000
Page 23
-------
Project and Category
Contact(s) Information Product Description
Environmental Monitoring Denice Shaw
for Public Access and
Community Tracking Charlotte
(EMPACT) Cottrill
statute of regulation
(Executive Order)
EMPACT sponsors and assists with local
projects that collect real-time and time-
relevant information about ambient
environmental conditions such as water
quality and ozone levels. At present, there
are EMPACT projects in 86 metropolitan
areas. See also http://www.epa.gov/empact/.
Index of Watershed
Indicators (IWI)
data management and
information dissemination
Charles IWI is a web-based product that displays
Spooner water quality data on regional and national
levels. The data displayed on the maps
reflect the current conditions and
vulnerability of water resources. IWI is
targeted at both states and the public. The
data come from a variety of sources
including OW partnerships with USGS and
NOAA. See also http://www.epa.gov/iwi/.
National Air Quality and
Emission Trends Report
statute or regulation
David Mintz The Air Quality Trends Report assesses
national and regional trends in air quality
Tom Curran and monitored emissions. OAR uses the
report to gauge air quality, pollution
reduction, and pollutant levels as a part of
the Clean Air Act (CAA). Sections include
discussion on air quality trends for criteria
pollutants, air quality on local levels, the
status of areas that do not meet NAAQS
standards, air toxics, visibility trends, and
acid deposition. The report is available in
hard copy or as a PDF file from the OAR
web site. See also http://www.epa.gov/oar/
oarpubs.html.
Pennsylvania Compliance Kim Nelson
Reporting System (PA CRS)
compliance focus
Pennsylvania designed PA CRS to collect and
display compliance information for
Department of Environmental Protection
(DEP) employees. As part of a shift in DEP
policy away from enforcement and towards
compliance assistance, PA CRS measures and
reports on ongoing compliance efforts. While
the reporting system itself is for internal use
only, results are reported on the web for the
public. See also http://falcon.state.pa.us/crs/
web/crs interface.frames start.
Page 24
November 17, 2000
-------
Project and Category
Contact(s) Information Product Description
Region III Enviroviz
data management and
information dissemination
Alvin Morris Enviroviz displays compliance, environmen-
tal quality, budget, and GPRA information
Rich Paiste on maps and graphs using two and three-
dimensional graphics. Users can "brush"
certain features to retrieve metadata about
the monitoring station, data collection
method, numerical data, specific pollutants,
and other details. Users can view informa-
tion by region, state, congressional district,
county, or watershed. Enviroviz is not yet
available beyond the pilot project.
Sector Facility Indexing
Project (SFIP)
compliance focus
Mike Barrette SFIP is a web-based pilot project that
integrates environmental compliance data
from five different industrial sectors includ-
ing automobile assembly, iron and steel
production, pulp manufacturing, petroleum
refining and nonferrous metals smelting and
refining. It reports on approximately 650
facilities. Its primary purpose is to centralize
and increase accessibility to facility informa-
tion from a range of EPA databases.
See also http://es.epa.gov/oeca/sfi/.
South Baltimore Pilot Project Henry Topper This project represents a community-based
approach to environmental protection and
stakeholder involvement economic development. When complete, it
will provide South Baltimore communities
with risk-based environmental information
on a variety of areas including cancer, parks,
abandoned housing and trash, economic
development, and air and water quality.
Volunteer teams of citizens and environmen-
tal, industry, and government representatives
coordinate different project components. See
also http://www.epa.gov/opptintr/cbep/
baltodex.htm.
Toxics Release Inventory
(TRI)
statute or regulation
Steven Mandated by the Emergency Planning and
Newburg- Community Right to Know Act (EPCRA) of
Rinn 1986, TRI displays information about
chemicals designated as potentially hazard-
Odelia Funke ous. Facilities that meet TRI requirements
must report on their estimated chemical
John Melone releases, transfers, waste treatment, pollution
prevention, and recycling activities. TRI data
are available in web and CD-ROM format as
well as in various information packets and
reports. See also http://www.epa.gov/tri/.
November 17, 2000
Page 25
-------
e AĞy,-v\nry C.ww^Pi'i'
Name
George Bonina
Office of Environmental Information (OEI)
Tom Curran
Office of Air and Radiation (OAR)
David Davis
Q^ce o/fFater (^O^
Steve Goranson
Region V
Barnes Johnson
Office of Solid Waste and Emergency Response
(OSWER)
Alvin Morris
Region III
Charles Spooner
Office of Water (OW)
Frederick Stiehl
Office of Enforcement and Compliance
Assurance (OECA)
Henry Topper
Office of Prevention, Pesticides, and Toxic
Substances (OPPTS)
Date Interviewed
August 30, 1999
August 12, 1999
August 18, 1999
September 28, 1999
August 24, 1999
Septembers, 1999
Septembers, 1999
August 26, 1999
September 13, 1999
Page 26
November 17, 2000
-------
Product Identification
What was the original impetus for developing your project?
What information need is the product designed to address?
Who is your primary audience? What methods, if any, did you use to
target your audience or get feedback during the development process?
and
Have you included stakeholder input in the development of your product?
Product Development
How much raw data are included in the project? How much interpreta-
tion? What kind of metadata do you make available to product users?
Are you concerned about the respectful use or misrepresentation of the
information included in your product? If so, what do you do to address
these concerns?
Do you solicit customer feedback? If so, how? Do you target a specific
subset of your users, or do you attempt to reach all users?
Do you make any effort to coordinate your product with others offered by
EPA through linkages, shared data, or another mechanism? Have you
tried to integrate your product with other databases or products?
Do you have a process for identifying or soliciting alternative reporting
methods for data? Do you accept data from external sources? and
Are there Agency guidelines that dictate how you collect or present data?
Do you submit your data or product to peer review?
Product Management and Future Commitment
What resources do you currently have available for the production of
your information product?
Do you have a product evaluation program in place? If so, who are the
evaluators? Has your product changed over time because of such input?
Do you think an evaluation process adds credibility to a product?
November 17, 2000 Page 27
-------
What kind of political commitment is there to sustaining your product in
your office/in the Agency? Do you think that the current level of commit-
ment will be sufficient to support the continued growth and development
of your product?
Do you expect any dramatic changes in your future mode of operation or
in the maintenance of your product; and
What would you do differently if you could redesign this product?
Page 28 November 17, 2000
------- |