Evaluation
of the Effectiveness of the Implementation Tools for the
National Emission Standards for Hazardous Air Pollutants
for Aerospace Manufacturing and Rework Facilities
Prepared for:
Tom Link
Information Transfer and Program Integration Division
Office of Air Quality Planning and Standards
U.S. Environmental Protection Agency
Prepared by:
EC/R Incorporated
April 29, 2004
-------
EXECUTIVE SUMMARY
The Program Implementation and Review Group (PIRG) within the Information Transfer
and Program Integration Division (ITPID) of the U.S. Environmental Protection Agency's (EPA)
Office of Air Quality Planning and Standards (OAQPS) has been developing implementation
tools for EPA's air quality regulations for years. Until now, PIRG has received only anecdotal
feedback on the usefulness and effectiveness of these tools from their target audiences (industry
and Federal, State, and local air pollution control agencies). There has been no structured
mechanism for soliciting feedback on these tools. The purpose of this evaluation was to conduct
a formal survey of the users of PIRG's implementation tools to determine the effectiveness of the
tools. The tools for the National Emission Standards for Hazardous Air Pollutants (NESHAP)
for Aerospace Manufacturing and Rework Facilities (Aerospace NESHAP) were chosen for this
pilot study because the numerous Aerospace NESHAP implementation tools are still largely used
as a model for other tools and because PIRG had a good list of the recipients and potential
recipients of the tools. This pilot study will be used as a template for future efforts to evaluate
the effectiveness of PIRG's implementation tools.
Out of the 135 stakeholders invited to take the Aerospace NESHAP implementation tools
effectiveness survey, 20 responses were received. Some of the major findings from these survey
responses include:
The Aerospace NESHAP implementation tools were used by the target audiences;
The target audiences found the Aerospace NESHAP implementation tools to be
helpful;
Some respondents among the target audiences did not know about the Aerospace
NESHAP implementation tools;
The Air Toxics Website and e-mail lists could be effective means of spreading
awareness about the tools;
The Aerospace NESHAP implementation tools are still being used seven years
after they were developed;
The Aerospace NESHAP implementation tools helped affected sources
understand how to submit reports and make process and equipment changes; and
The Aerospace NESHAP implementation tools made a difference in reducing air
pollution.
These and other lessons learned from this survey will be used by PIRG to develop more useful
implementation tools. In fact, some of these lessons are already being used. For example, the
survey indicated that the applicability flowcharts for the Aerospace NESHAP were not very
useful, which was the result PIRG expected. PIRG has been using a different format for
applicability flowcharts developed subsequent to the Aerospace flowcharts after realizing that the
format was confusing and not user-friendly. The survey results confirm that changes PIRG has
made to implementation tools since those for the Aerospace NESHAP were developed will make
the tools more useful to the target audiences.
11
-------
One aspect of this evaluation that needs further examination before future efforts are
undertaken is the low response rate. A strategy should be devised for increasing the response rate
on future surveys without violating anyone's privacy or relying too heavily on trade associations
for lists of stakeholders.
Finally, the success of this pilot study indicates that an effectiveness survey should
become a standard aspect of the process for developing implementation tools for OAQPS'
regulations.
111
-------
TABLE OF CONTENTS
EXECUTIVE SUMMARY ii
I. INTRODUCTION 1
II. IMPLEMENTATION TOOL DEVELOPMENT AND DISTRIBUTION 2
A. Tool Development 2
B. Tool Distribution 3
C. Tool Feedback 3
III. TOOL EVALUATION SURVEY 4
A. Survey Development 4
B. Survey Distribution 6
IV. SUMMARY OF SURVEY RESULTS 7
A. The Respondent Population 7
B. Access to the Implementation Tools 8
C. Rated Utility of the Implementation Tools 10
D. Rated Utility of Implementation Tools for Increasing Understanding of How to
Meet NESHAP Requirements 14
E. Changes to the Implementation Tools 17
V. LESSONS LEARNED 19
A. Developing and Improving NESHAP Implementation Tools 19
B. Designing Online User Surveys 21
1. Office of Management and Budget (OMB) Survey Approval Process . . 21
2. Coordinating with National Technology Services Division 22
3. Technical Issues 23
4. Communicating with Stakeholders 25
VI. CONCLUSIONS AND RECOMMENDATIONS 27
Attachment 1: The Aerospace Implementation Tools Evaluation Survey Al-1
Attachment 2: Invitation Template for Beta-test Group A2-1
Attachment 3: Invitation Template for Stakeholders A3-1
Attachment 4: Security Certification Form A4-1
iv
-------
FIGURES
Figure 1. Monthly timeline of major steps taken to conduct the online survey 5
TABLES
Table 1. The survey population 8
Table 2. Why did you not obtain copies of any of the implementation tools 9
Table 3. How did you obtain the implementation tools for the Aerospace NESHAP? .... 9
Table 4. When did you obtain the implementation tools for the Aerospace NESHAP? . . 10
Table 5. Have you used any of the implementation tools for the Aerospace NESHAP? . 10
Table 6. Why did you not use any of the implementation tools for the Aerospace
NESHAP? 11
Table 7. Please rate the usefulness of each of the implementation tools for the Aerospace
NESHAP that you have used 12
Table 8. I used this tool for the following purposes 13
Table 9. Are the implementation tools for the Aerospace NESHAP still useful to you now?
14
Table 10. The implementation tools for the Aerospace NESHAP increased my
understanding of how to carry out the following regulatory actions 15
Table 11. The implementation tools for the Aerospace NESHAP increased my
understanding of how to make the following process and equipment changes . . 15
Table 12. The implementation tools for the Aerospace NESHAP increased my
understanding of how to carry out the following management actions 16
Table 13. Do you think your use of the implementation tools and your increased
understanding of the Aerospace NESHAP have resulted in pollutant reductions at
your facility? 16
Table 14. Do you think your use of the implementation tools and your increased
understanding of the Aerospace NESHAP have resulted in cost savings? 17
v
-------
EVALUATION OF THE EFFECTIVENESS OF THE AEROSPACE NESHAP
IMPLEMENTATION TOOLS
I. INTRODUCTION
In 1997, the Program Implementation and Review Group (PIRG) within the Information
Transfer and Program Integration Division (ITPID) of the U.S. Environmental Protection
Agency's (EPA's) Office of Air Quality Planning and Standards (OAQPS) developed a suite of
implementation tools for the National Emission Standards for Hazardous Air Pollutants
(NESHAP) for Aerospace Manufacturing and Rework Facilities (Aerospace NESHAP). The
Aerospace NESHAP implementation tools were designed to help industry understand and
comply with the Aerospace NESHAP and to help regulators understand and ensure that sources
comply with the NESHAP. The tools convey basic information about the Aerospace NESHAP
through summaries, flowcharts, checklists, and other documents created with the intent of being
easy for stakeholders to use. The tools also include example reporting and recordkeeping forms
with example responses. The tools were distributed to numerous aerospace stakeholders
(affected sources, trade associations, Regional offices, State and local agencies, contractors, etc.)
through mail and the Internet. At the time of their development, the Aerospace NESHAP
implementation tools were considered the most comprehensive set of tools available, and were
subsequently used as a model for developing implementation tools for other NESHAPs.
In order to make the most appropriate use of the available resources for implementation
efforts for other NESHAPs and for residual risk standards, PIRG conducted an evaluation of the
effectiveness of the Aerospace NESHAP implementation tools. Although many lessons have
been learned since the development of these tools and some changes have already been made to
the format and content of subsequent sets of implementation tools, the Aerospace NESHAP tools
were chosen for this evaluation because there was a detailed record of people to whom the tools
were or may have been distributed and because a wide variety of tools was developed for this
regulation. This evaluation consisted of a survey designed to assess how useful the Aerospace
NESHAP implementation tools were to members of the intended audience in helping them
understand the rule and comply with or enforce it. The evaluation constitutes the first phase of a
project aimed at improving the design, distribution, and effectiveness of implementation tools.
Using the results of the Aerospace NESHAP implementation tools evaluation study, PIRG
intends to improve existing and future tools as well as develop a template for conducting
evaluations of other implementation tools.
This report presents the results of the Aerospace NESHAP implementation tools
evaluation study. The report includes a summary of the Aerospace NESHAP implementation
tools, a description of how the survey was developed and distributed, a summary of the responses
received, and conclusions that can be drawn about the effectiveness of the Aerospace NESHAP
implementation tools. This report also includes a discussion of lessons that were learned during
1
-------
the evaluation process about both improving NESHAP implementation tools and designing
online user surveys.
II. IMPLEMENTATION TOOL DEVELOPMENT AND DISTRIBUTION
This section provides a brief description of the Aerospace NESHAP implementation
tools, how they were developed and distributed, and what feedback was received regarding their
usefulness prior to this evaluation.
A. Tool Development
The implementation tools for the Aerospace NESHAP (tools) were developed following
rule promulgation in 1995 under the lead of Ingrid Ward (PIRG). Individuals from the Federal
and Regional levels of EPA, along with two State and local representatives, worked closely with
stakeholders to develop a suite of tools that would be useful to both industry and regulators.
Stakeholders were given the opportunity to review and comment on draft versions of the tools
before the final versions were prepared. The end product was a comprehensive set of tools which
became the model for developing implementation tools for other NESHAPs. The tools that were
developed during this process and that are currently available on EPA's Air Toxics Website
include the following:1
Overview Brochure: a brochure containing a compliance schedule, summary of rule
requirements, and EPA's Regional office contacts (9 pages)
Rule Summary: an implementation document containing a compliance timeline,
regulatory overview, applicability flowcharts, and inspection checklist (130 pages)
Recordkeeping Forms: various example recordkeeping forms including a blank form
and example response
Calculations: example calculations (5 pages)
Reporting Forms: example reporting forms with example responses for initial
notification, notification of compliance status, and semiannual notification
Frequently Asked Questions (FAQs): a compilation of Questions and Answers which
are numbered and organized by topic (117 Q&A's, 27 pages)
'httpV/www.epa.gov/ttn/atw/aerosp/aeropg.htmlSIMP
2
-------
Applicability Guidance: applicability guidance memoranda from EPA Region 10 (2
memos, 3 pages)
List of Manufacturers: a list of manufacturers of Method 319 paint arrestors (2 pages)
B. Tool Distribution
According to a 1992 survey of the aerospace industry conducted during the rule
development process, there were approximately 418 sources potentially subject to the Aerospace
NESHAP. Subsequent interaction with the industry indicates that this estimate is high, but no
other estimate is readily available. In addition to these potentially affected sources; Regional,
State, and local regulators were among the intended recipients of the Aerospace NESHAP
implementation tools.
The implementation tools were primarily distributed through the Aerospace page on the
Unified Air Toxics Website (UATW, now the Air Toxics Website).2 The tools were posted on
this page in March 1998, and interested parties were able to download copies of the tools from
this website. According to archived usage reports for the UATW,3 the Aerospace page received
an average of 65 hits and 55 downloads per week through the end of 1999. The Aerospace page
was among the top 50 hits and the top 50 downloads among UATW sites almost every week
from March 1998 through December 1999.
Although there was no formal direct mailing, approximately 500 hard copies of the
original implementation document were printed. Some hard copies were sent to EPA's library
system, from which interested parties could request hard copies. Individual hard copies were
mailed to interested parties upon request. Additional hard copies, above the original 500, were
distributed at regional training sessions which were conducted in August 1998. Training sessions
of varying lengths (half-day up to two days) were conducted in Newington, Connecticut; Cocoa
Beach, Florida; Kansas City, Kansas; Seattle, Washington; and San Francisco, California with
the assistance of Regional office staff. These training sessions were attended by a total of
approximately 400 people representing Regional offices, State agencies, local agencies, sources,
trade associations, and contractors and were conducted for the purpose of providing a better
understanding of the regulatory requirements of the Aerospace NESHAP.
C. Tool Feedback
Since the development of the tools, Ms. Ward has occasionally received informal
feedback on their usefulness. Ms. Ward reported that the question and answer document, in
particular, has been extremely helpful in reducing the amount of time that Emissions Standards
2
http ://www.epa.gov/ttn/atw/aerosp/ aeropg.html
3
These reports are archived on EPA's Intranet and are not available to the public.
3
-------
Division (ESD) and ITPID staff spend answering questions from stakeholders. Originally, these
questions and answers were to be compiled into a searchable database, but the database was not
properly designed and so proved to be unhelpful. Additionally, Ms. Ward has received very
positive feedback from both industry and regulators on the example reporting forms. A formal
study had never been conducted to determine whether these tools were effective and suitable as a
model for other rules.
III. TOOL EVALUATION SURVEY
This section outlines the survey development process as well as to whom and how the
survey was distributed. A monthly timeline of the major steps taken to develop the survey is
included as Figure 1.
A. Survey Development
A formal study to evaluate the effectiveness of the Aerospace NESHAP implementation
tools was undertaken by PIRG under the lead of Tom [.ink. After consideration of a variety of
data-gathering methods, PIRG decided to conduct the evaluation of the implementation tools
using an online survey. The ease of distributing and responding to an online survey was
considered advantageous for this evaluation.
The survey questions were developed by closely following the guidance of EPA's
Compliance Assistance Generic Information Collection Request (ICR).4 The questions were
directed at determining whether stakeholders knew about, obtained, and used the implementation
tools; how useful they found the tools, whether the tools were helpful for compliance, and how
they would improve the tools. While the survey was completely anonymous and EPA had no
way of associating a survey response with a respondent, a demographic question was used in the
survey to determine whether each respondent was affiliated with a source, governmental agency,
or other organization. A copy of the survey questions (not in the online format) is included as
Attachment 1 to this report. In the online version, only the questions relevant to each respondent
(based on the answers they provided) appeared on the screen. Additional details about the ICR
and the development of the survey questions are provided in a separate document.5 Lessons
learned from the ICR approval process are described in Section V.B.I.
The format and function of the survey web page was created through the combined efforts
of both those involved in the development of the implementation tools and the evaluation study
and personnel from the Office of Enforcement and Compliance Assistance (OECA), who had
previously developed a similar type of survey. The web page consisted of an introduction to the
4http ://www.epa.gov/icr/icrs/l 860 02 .html
5Deering, Ann. "Evaluation Study Design to Determine the Effectiveness of the Aerospace NESHAP Implementation
Tools." Prepared by EC/R Incorporated, prepared for Information Transfer and Program Integration Division, U.S.
Environmental Protection Agency. Sept 2003.
4
-------
October
© Submit survey to the Office of Management and Budget (OMB)
© Begin designing the back-end database
© Begin designing the online survey
November
© Prepare information needed for the Application Deployment Checklist
© Obtain approval of the survey from the Office of Management and Budget
December
© Post a draft survey and back-end database on EC/R's server for EPA comment
and review
© Post a draft survey on EPA's Intranet for technical testing
© Obtain approval from the National Technology Services Division (NTSD) to
post the survey on EPA's public server
January
© Contact relevant trade associations to request aid in distribution of the survey
© Prepare survey distribution lists
© Conduct a beta-test of the survey on EC/R's server with help from selected
stakeholders
February
© Make final edits to the survey and database
© Notify NTSD that the survey is ready for final review and production on EPA's
public server
March
© Survey period
April & May
© Review and analyze survey results
Figure 1. Monthly timeline of major steps taken to conduct the online survey.
5
-------
survey and a series of questions. The survey was formatted so that, as a respondent provided
answers, additional response-specific questions would appear on the screen. At the end of the
survey, the respondent was asked to submit their responses to EPA by clicking on a "submit"
button. Respondents were able to alter their answers until they clicked "submit." The final
version of the survey was developed using Domino code. Answers were stored in a back-end
database which could be exported to a single table in Lotus 1-2-3.
A beta-test version of the online survey was created and hosted on EC/R Incorporated's
server. The survey web page explained the purpose of the evaluation and presented a beta-test
version of the survey along with links to the Aerospace NESHAP implementation tools. The
beta-test was conducted on EC/R's server rather than EPA's to avoid delays in the project
timeline associated with meeting EPA requirements for posting surveys on the public server.
Using this strategy, PIRG was able to conduct the beta-test while continuing to take the necessary
steps for posting the final version of the survey on EPA's public server. Invitations were sent by
Mr. [.ink to a beta-test group of 18 stakeholders identified by Ms. Ward as active participants in
aerospace-related activities. In addition to responding to the beta-test of the survey for technical
testing purposes, the beta-test group was asked to comment on the survey questions and the
function of the survey, and to provide contact information for any additional stakeholders. The
invitation that was sent to the beta-test group is included in this report as Attachment 2. The few
comments that were received during the beta-test period were incorporated into the final version
of the survey. No contact information for additional stakeholders was received.
After developing the final version of the survey and ensuring the back-end database was
correctly capturing responses, the survey went through final staging review by the Application
Deployment Checklist (ADC) coordinators of EPA's National Technology Services Division
(NTSD). The survey was approved by the ADC coordinators and posted on EPA's public server.
The ADC coordinators also implemented the tracking of the number of hits the survey received
while on EPA's public server.
B. Survey Distribution
Invitations to participate in the survey were directly sent to 199 people including
representatives of industry, consulting firms, and Federal, State, and local government agencies,
including the military and EPA. Only 135 of the individuals who were sent invitations to
participate in the survey were expected to be members of the target audience for the Aerospace
NESHAP implementation tools. These 135 individuals are considered the stakeholder group for
the purpose of calculating the survey response rate. The other 64 individuals who were sent an
invitation to the survey are EPA staff who would have an interest in the evaluation process and
results but were not expected to provide a response to the survey.
In the invitation, the stakeholders were encouraged to forward information about the
survey to any other relevant individuals. The survey was located on EPA's public access server,
and no links to the survey were available from any other pages. It was assumed that people who
6
-------
accessed the survey were either directly invited or received a forwarded invitation from someone
who was directly invited to participate in the survey. The group of 135 stakeholders was
identified with the help of Ms. Ward and Robert Peters, Director of Environmental Safety and
Health for the Aerospace Industry Association (AIA). The invitations were sent to stakeholders
by Ms. Ward and to appropriate members of the AIA by Mr. Peters. While both Ms. Ward and
Mr. Peters adjusted the invitation to fit their needs, the basic invitation format is included as
Attachment 3 to this report. The survey was completely anonymous, therefore a list of
stakeholders to whom direct invitations were sent is not included in this report. Initial invitations
were sent by e-mail on March 1 & 2, 2004. A reminder e-mail was sent by both Ms. Ward and
Mr. Peters to their respective lists on March 22 & 23, 2004. Of the 135 stakeholders, 14
stakeholders received the initial invitation but not the reminder e-mail, and 12 stakeholders
received the reminder e-mail but not the initial invitation. Of the 199 people invited to
participate in the survey, 70 were invited by Mr. Peters, 125 were invited by Ms. Ward (including
the 64 non-stakeholders), and 4 were invited by both Ms. Ward and Mr. Peters.
IV. SUMMARY OF SURVEY RESULTS
This section presents and summarizes the responses that were received during the survey
period. The stakeholders were asked to respond to the survey by March 31, but results were
collected and analyzed through April 2. The titles for most of the following tables correspond
with the questions asked in the survey. The first column represents the possible responses that
could have been chosen by the respondent.
A. The Respondent Population
Direct invitations to participate in the survey were sent to a total of 199 people, 135 of
whom were considered stakeholders. These stakeholders were also encouraged to forward the
invitation to other stakeholders. A total of 20 responses were received. This is a response rate of
approximately 10 percent of all the invitees or 15 percent of the stakeholder invitees. The ICR
that was submitted for this survey projected a response rate of 15 percent, which was achieved.
Statistics were collected during the survey period to track the number of hits the survey
received. The statistical package used tracks each time an individual file on the survey page is
accessed. The survey page contains multiple files, such as the EPA logo on each page. This
means that for every one time a respondent accesses the survey, multiple hits may be registered.
By dividing the total number of hits by the number of files, it was estimated that the survey was
accessed approximately 30 times during the period from March 1 through 31. Since EC/R staff
accessed the site for testing purposes several times throughout this period, and 20 responses were
received during this period, it was assumed that most of the stakeholders who accessed the
survey also provided a response to it.
7
-------
Table 1. The survey population
Stakeholder Type
Number of Respondents
Percent of Total Respondents
A I'lecled Source
6
30
Federal (io\ eninieiil
5
25
Siale (io\ eninieiil
1
5
Local (io\cmmciU
5
25
Trade Association
1
5
(,'oiiMilling Firm
1
5
I.aw Firm
0
0
Oilier*
1
5
TOTAL
20
100
The respondent who marked 'other' did not provide any specification as to the organization with which
he/she is affiliated.
The affected source group was the most well-represented type of organization among the
survey respondents. Federal and local government representatives made up the two other groups
that provided the most responses. Eighty-five percent of the respondents were from the target
audiences for the Aerospace NESHAP implementation tools (affected sources and Federal, State,
and local governments). The only type of organization not represented by any respondents was
law firms.
B. Access to the Implementation Tools
A total of 17 respondents, or 85 percent of the total respondents,6 indicated that they
obtained copies of the implementation tools. Table 2 summarizes the reasons stakeholders
indicated they did not receive the tools. Table 3 summarizes how those stakeholders that did
obtain the tools obtained them, and Table 4 summarizes when the respondents obtained the tools.
6 This group of 17 respondents was not the same as the group of 17 respondents who represented the target audiences
for the Aerospace NESHAP implementation tools.
8
-------
Table 2. Why did you not obtain copies of any of the implementation tools?
Response
Nilmher of
Respondents
Percent out of the
Respondents who Didn't
Obtain Tools
Percent of
Total
Respondents
1 didn't know the\ existed
2
67
10
1 am not the appropriate
person to use lliem
1
33
5
1 tried but was unable to
obtain copies
0
0
0
Other
0
0
0
TOTAL
3
100
15
Table 3. How did you obtain the implementation tools for the Aerospace NESHAP?
(respondents could choose more than one means of obtaining the tools)
Response
Nilmher of
Respondents
Percent out of the 17
Respondents who
Obtained Tools
Percent of
Total
Respondents
l-l'.Vs Air Toxics Website
12
7|
60
1-1'A's Compliance Assistance
Clearinghouse
1
h
5
Hard cop\ direct through the mail
1
h
5
Training session or conference
J
\2
10
Colleague ga\e me a cop\ link
5
21-)
25
Other*
i
h
5
The respondent who marked "other" indicated that he/she received the implementation tools through direct
involvement in the rulemaking process.
9
-------
Table 4. When did you obtain the implementation tools for the Aerospace NESHAP?
(respondents could choose more than one option)
Response
NilmIter of
Respondents
Percent out of the 17
Respondents who
Obtained t ools
Percent of Total
Respondents
IJ\ llie compliance dale
4
24
20
Alter llie compliance dale
4
24
20
1 don"l remember
j
12
10
Some lool> l\\ llie compliance
dale iiic lool> alter
6
35
30
According to survey responses, the majority of respondents obtained at least some of the
tools before the compliance date. The Air Toxics Website was the primary means by which tools
were obtained. Those respondents that did not obtain copies of the tools indicated that they
either did not know that the tools existed or that they are not the appropriate person to use them.
C. Rated Utility of the Implementation Tools
A total of 17 respondents, or 85 percent of the total respondents, indicated that they
received the implementation tools for the Aerospace NESHAP. In this section we summarize
whether those respondents used the tools and, if so, how useful they found them. Table 5
summarizes how many people who received the tools used them. Table 6 summarizes why the
respondents who received and did not use the tools did not use them. Table 7 summarizes the
utility of each implementation tool to the respondents who used them. Table 8 summarizes what
the government and consultant respondents that used the implementation tools used them for,
and Table 9 summarizes whether the tools are still useful to the respondents.
Table 5. Have you used any of the implementation tools for the Aeros
nac-e NESHAP?
Response
Number of Respondents
Percent out of the 17
Respondents w ho
Obtained Tools
Percent of Total
Respondents
Ye>
16
94
80
No
1
6
5
10
-------
Table 6. Why did you not use any of the implementation tools for the Aerospace NESHAP?
Response
Number of
Respondents
Percent out of the 17
Respondents who
Obtained Tools
Percent of
I'otal
Respondents
1 received llicm loo laic
0
0
0
The\ were loo difficult lo
undciMand
0
0
0
1 \\a> nol llic appropriate person
lo ha\c received llic lool*
0
0
0
The loo 1 were nol helpful lo me
0
0
0
Oilier*
1
h
5
TOTAL
1
b
5
The respondent who marked "other" indicated that he/she did not use the tools because they were very long, and
7
noted that there were 123 pages.
7
This is approximately the length of the "Summary of Requirments" doccument which includes many of the
implementation tools in one document. PIRG has since decided to make each tool a separate item for download and distribution
to decrease the length of each individual tool and to make it easier for stakeholders to obtain only the tools they need.
11
-------
Table 7. Please rate the usefulness of each of the implementation tools for the Aerospace
NESHAP that you have used.*
Tool
Percent out of the 16
Respondents who used any
implementation tools
Percent of Tohil Respondents
Very Good
Good
Satisfactory
Poor
Very Poor
Didn't Use
Very Good
Good
Satisfactory
Poor
Very Poor
Didn't Use
Overview Brochure
25
38
13
0
0
25
20
30
10
0
0
20
Compliance Timeline
25
31
13
0
0
31
20
25
10
0
0
25
Regulatory Overview
31
38
6
0
0
25
25
30
5
0
0
20
Applicahility F1 owcharts
19
25
19
0
0
38
15
20
15
0
0
30
Inspection Checklists
19
56
6
0
0
19
15
45
5
0
0
15
Example Recordkeeping
Forms
31
38
6
0
0
25
25
30
5
0
0
20
Example Calculations
13
38
6
0
0
44
10
30
5
0
0
35
Example Reporting Forms
31
38
6
0
0
19
25
30
5
0
0
15
Frequently Asked Questions
50
19
13
0
0
19
40
15
10
0
0
15
Applicability Ciuidance
Memos
25
31
6
0
0
38
20
25
5
0
0
30
List of Manufacturers of
Method 319 Paint Arrestors
31
38
0
0
0
31
25
30
0
0
0
25
All 16 respondents who used some of the implementation tools either rated each of the tools or indicated that
they did not use the tool.
12
-------
Table 8. I used this tool for the following purposes. (Note that this question was only asked of
government iiud eonsiilhinl respondents iiud multiple purposes could /v chosi-n.)
Tool
Percent out of the
(ov*t/CoiiMilhint Respondents who
used any implementation tools
Percent of Total
C>ov*t/Consultant
Respondents
Compliance
Assistance
Permitting
Enforcement
Compliance
Assistance
Permitting
Enforcement
Overview Brochure
78
0
56
58
0
42
Compliance Timeline
56
0
33
42
0
25
Regulatory Overview
67
11
44
50
8
33
Applicability Flowcharts
56
11
33
42
8
25
Inspection Checklists
56
0
67
42
0
50
F. xa mp 1 e Recordkeeping
Forms
56
0
56
42
0
42
F.xample Calculations
44
0
33
33
0
25
F.xample Reporting Forms
67
0
44
50
0
33
Frequently Asked Questions
78
11
44
58
8
33
Applicability Ciuidance
Memos
56
11
33
42
8
25
List of Manufacturers of
Method 319 Paint Arrestors
56
0
22
42
0
17
13
-------
Table 9. Are the implementation tools for t
le Aerospace NESHAP still useful to you now?
Response
NilmIter of
Respondents
Percent out of the 16
Respondents who I sed
the Tools
Percent of Total
Respondents
Ye>
14
88
70
No. llie\ arc not up lo dale
0
0
0
No. llie\ are nol umMiiI lor
maintaining compliance
0
0
0
No. m\ job no longer
require^ me lo u>e lliem
2
13
10
Oilier
0
0
0
While almost all of the respondents that obtained copies of the implementation tools used
at least one of them, many respondents did not use all of them. The respondents indicated that
most of the tools they did use were either "good" or "very good." Some tools were considered
"satisfactory," but none were considered "poor" or "very poor" by any of the respondents. The
Frequently Asked Questions document received the best overall rating. The Regulatory
Overview, Example Recordkeeping Forms, Example Reporting Forms, and List of
Manufacturers of Method 319 Paint Arrestors also received high ratings. The Example
Calculations, Applicability Flowcharts and Applicability Guidance Memos received the lowest
totals of "very good" and "good" responses and were also used the least of all of the
implementation tools. The Inspection Checklists, Example Reporting Forms, and Frequently
Asked Questions were the most used implementation tools. Government and consultant
respondents indicated that they used the tools mainly for compliance assistance or enforcement.
Only four of the tools were used for permitting (Regulatory Overview, Applicability Flowchart,
Frequently Asked Questions, and Applicability Guidance Memos). Most respondents indicated
that the tools are still useful to them now.
D. Rated Utility of Implementation Tools for Increasing Understanding of How to
Meet NESHAP Requirements
This section summarizes how useful the implementation tools were at helping affected
sources carry out the requirements of the Aerospace NESHAP. The following tables summarize
the usefulness of the tools at helping affected sources understand how to carry out regulatory
actions, make process and equipment changes, and carry out management actions. The last two
tables of this section summarize whether increased understanding of the NESHAP through use of
the implementation tools resulted in pollutant reductions and cost savings, respectively, at
affected sources. All six of the affected source respondents indicated that they used the
implementation tools.
14
-------
Table 10. The implementation tools for the Aerospace NESHAP increased my understanding of
how to carry out the following regulatory actions (respondents could choose more than one
Response
NilmIter of
Respondents
Percent out of the 6 Affected
Source Respondents
Submitting a Notification
5
83
Submitting Report*
h
100
Submitting 1 )ata
4
67
Obtaining a Permit
->
The implementation tool* were not
help In 1
0
0
Other
0
0
Table 11. The implementation tools for the Aerospace NESHAP increased my understanding of
how to make the following process and equipment changes {respondents could choose more than
one change):
Response
Nil ill Iter of
Respondents
Percent out of the 6 Affected
Source Respondents
(.'hanging llie storage or handling of
a waste or emission
->
(.'hanging a process or practice
5
83
Implementing;! material or waste
rec\ cling sWem
0
0
Installing new process equipment
4
67
Installing pollution control
equipment
4
67
The implementation tools were not
helpful
0
0
Other
0
0
15
-------
Table 12. The implementation tools for the Aerospace NESHAP increased my understanding of
how to carry out the following management actions (respondents could choose more than one
Response
NilmIter of
Respondents
Percent out of the 6 Affected
Source Respondents
Instituting naming or other
communication u> improve
5
83
auaienc^ and or practices
1 \stahlishing an cm iionnienlal
management sWeni
50
Conducting a >el l-audit
50
The implementation tools were not
helpful
i
17
Other
0
0
Table 13. Do you think your use of the implementation tools and your increased understanding
of the Aerospace NESHAP have resulted in pollutant reductions at your facility?
Response
Nil ill Iter of
Respondents
Percent out of the 6 Affected
Source Respondents
Yes
4
67
No
->
I'm not miiv
0
0
There ha\e not been pollutant
reductions
0
0
TOTAL
h
100
Affected source respondents provided the following details regarding pollutant reductions
at their facilities:
"Air emissions reduction of 52% TCE consumption from vapor degreaser."
"We have recently adopted various zero-VOC primers and topcoats, reducing annual
emissions by several tons per year."
16
-------
Table 14. Do you think your use of the implementation tools and your increased understanding
of the Aerospace XESIIAP have resulted in cost savings?
Response
NilmIter of
Respondents
Percent out of the 6 Affected
Source Respondents
Ye*
->
No
1
17
I'm not miiv
50
There lia\e not been co*l *a\ing*
0
0
TOTAL
h
100
All of the survey respondents who are affected sources indicated that the implementation
tools increased their understanding of how to submit reports. Most of the affected sources also
indicated that the tools increased their understanding of how to change a process or practice and
how to institute training or other communications to improve awareness and/or practices. A
majority of affected source respondents (67 percent) indicated that using the implementation
tools increased their understanding of the Aerospace NESHAP and resulted in pollutant
reductions at the facility. A variety of responses were received to the question regarding cost
savings, and no respondents provided any details on cost savings associated with implementing
the NESHAP.
E. Changes to the Implementation Tools
This section summarizes changes that respondents made to the tools and how they
suggested EPA could improve the tools. Seven, or 44 percent, of the 16 respondents who used
the implementation tools modified them to suit their needs. These respondents modified the
tools in the following ways:
"Modified the checklist to fit our processes"
"Developed our own specific checklist for spraybooths"
"Added more detailed instructions to the recordkeeping forms"
"Used the requirements in local procedures"
"Deleted sections of the reporting form that weren't applicable"
During the beta-test of the survey, we received the following additional details about
modifications that were made to the implementation tools:
"Some facilities used recordkeeping and notification forms as a model so they likely
made some changes to accommodate unique situations and facility specific
information"
17
-------
"The certification report was modified to only include the applicable activities at our
faciltiy"
"Streamlined directions for calculations"
"We developed our own inspection checklist and recordkeeping and reporting forms.
When the implementation tools became available we ensured our previously]
developed documents were adequate."
"Used them to develop compliance and operational checklists"
Respondents provided the following suggestions to improve the implementation tools:
"Inspection checklist tool needs to be more user friendly"
"[Make implementation tools] more to the point"
"[Provide] more answers to frequently asked questions..."
"Group Frequently Asked Questions so that they are easier to search"
"[Provide] more applicability guidance memos"
"Get tools on the website as soon as possible after the rule is promulgated"
Because of a recent inquiry from a minor source aerospace facility about daily
recordkeeping, one respondent looked closely at the implementation tool titled
"Example 4(a)- Primer and Topcoat Recordkeeping Form" and noted that:
the form needs to be provided in Excel format
a column is needed for total material applied in gallons, a sum of paint and
thinner
two columns are needed for daily VOC and HAP emissions in pounds per day
a form is also needed for multi- (two and three) part coatings
During the beta-test of the survey, the following suggestions were made to improve the
implementation tools:
"Make them available as soon as possible."
"Have them developed six months to a year prior to the final compliance date of the
rule."
"Ensure they are updated to reflect amendments and correct errors (Ingrid has been
very responsive in ensuring this has been done)."
"The tools I use most often are the frequently asked questions and guidance memos.
Frequent updates/additions to these tools are most useful. This helps maintain
consistency with how questions are answered for sources throughout the count[y]."
"On implementation tools such as inspection checklists, I would suggest striving to
make these checklists as concise as possible. Concise tools are easiest to use in the
field."
"Maybe some additional QA/QC. Found typos and such."
"I thought they were great in 1998 and still use them frequently. I can't think of any
changes."
18
-------
V. LESSONS LEARNED
A. Developing and Improving NESHAP Implementation Tools
Constructive feedback was received regarding the effectiveness of the implementation
tools developed for the Aerospace NESHAP. The responses received to this survey will be
useful in both improving the Aerospace NESHAP implementation tools and developing similar
future tools. Rather than reiterate the contents of each table included in the report, this section
includes a few highlights of the lessons learned from the survey that will be used to develop and
improve NESHAP implementation tools.
While the majority of respondents obtained copies of the tools and used them, there were
some respondents who did not know the tools existed. Efforts could be made to improve public
awareness of the tools and increase tool distribution. The Air Toxics Website was used by most
respondents to access the tools. One lesson learned is to increase awareness of the tools by
increasing awareness of the Air Toxics Website since it has been an effective means of tool
distribution.
The total lack of "poor" and "very poor" responses for any of the tools indicates that, in
general, EPA is developing useful implementation tools. The Frequently Asked Questions
document was both the most used and the most useful of all the tools. The comments received
from some of the respondents reiterate this point and suggest making it even more user-friendly
by, for example, organizing the questions by topic.8 The Inspection Checklists and Example
Reporting Forms were also among the most used tools, and the Regulatory Overview, Example
Recordkeeping Forms, and Example Reporting Forms were among the most useful tools. A
lesson learned here is to focus implementation efforts on FAQs, example forms and checklists,
and a regulatory overview. This confirms the anecdotal evidence Ms. Ward had that the example
reporting forms were well used and very useful. On the other end of the spectrum, the Example
Calculations, Applicability Flowcharts, and Applicability Guidance Memos were the least used
tools and were also among the least useful tools. If EPA is interested in preparing similar tools
in the future, it would be worth investigating why these three tools in particular were not well
used. Regarding the applicability flowcharts, shortly after they were developed, PIRG personnel
realized that the format was confusing and not helpful to the target audiences and, thus, began
using a different format for applicability flowcharts developed for other rules. Therefore, the
survey results with respect to the applicability flowcharts were expected and, in fact, confirm a
specific lesson already learned about developing implementation tools.
g
One suggestion obtained during the survey development process (not from the beta-testers or survey
respondents) was that the searchable database of comments received on the Office of Air and Radiation (OAR)
website, along with EPA's responses to these comments, which has been successfully implemented by OAR could
serve as a model for creating searchable databases of frequently asked questions (and their answers) for NESHAP
implementation tools.
19
-------
Most respondents indicated that the implementation tools for the Aerospace NESHAP are
still useful to them now, which emphasizes the suggestions that the tools should be kept accurate
and up-to-date. Many users of the tools are modifying them to suit their needs; the lesson learned
here is that the tools should be posted on the Air Toxics Website in both an easily-downloadable
format (such as .pdf) and an easily modifiable format (such as MS Word).
Affected source respondents indicated that the tools were useful in helping them
understand how to take regulatory actions, make process and equipment changes, and make
changes in management actions. In fact, 100 percent of the affected source respondents indicated
that the tools helped them understand how to submit reports, and 83 percent of the affected
source respondents indicated that the tools helped them understand how to change a process or
practice and how to institute training or other communication. Furthermore, 67 percent of the
affected source respondents indicated that the tools resulted in pollution reduction at their
facilities. The tools were least helpful in providing affected sources with an increased
understanding of how to obtain a permit, change the storage or handling of a waste or emission,
and implement a material or waste recycling system.
Government and consultant respondents indicated that they used the tools mainly for
compliance assistance and enforcement purposes and less so for permitting. Looking specifically
at the inspection checklists, PIRG expected to see them well used for compliance assistance and
enforcement, which the survey confirmed. PIRG also expected to see the FAQs well used for
enforcement, which the survey indicated was not the case. Overall, PIRG expected to see more
use of the tools for permitting purposes. One lesson learned is to target permitters when
publicizing the existence of implementation tools, perhaps through an e-mail listserv or other
mailing list. Another lesson learned is to review the tools to determine whether they could be
made more useful for permitting purposes.
Overall, respondents suggested that the tools be made available as soon as possible, and
specifically at least six months prior to the final compliance date of the NESHAP. Respondents
also suggested that tools be made concise and user-friendly and that they be kept up-to-date.
Several lessons that were learned regarding the development and improvement of
implementation tools include:
Increase awareness of the Air Toxics Website to publicize the availability of tools;
Focus implementation efforts on developing Frequently Asked Questions, example
forms, checklists, and a regulatory overview document;
Investigate why example calculations, applicability flowcharts and applicability
guidance memoranda are not well used;
Keep tools accurate and up-to-date;
Provide tools in formats that are both easy to download and easy to modify;
Include permitters when distributing tools;
Review tools for their usefulness to permitters;
20
-------
Develop tools that are concise; and,
Make tools available to the public in a timely fashion.
These lessons learned will be used to improve the development and distribution of
implementation tools in the future. In fact, some of these lessons learned are already being
implemented by PIRG.
B. Designing Online User Surveys
1. Office of Management and Budget (OMB) Survey Approval Process
In order to survey more than nine people, the survey needed to be approved by the Office
of Management and Budget (OMB). To expedite the approval process for the Aerospace
NESHAP Implementation Tools survey, PIRG made use of the generic Compliance Assistance
Information Collection Request (ICR), with EPA ICR No. 1860.02, OMB Control No. 2020-
0015, and a full title of "Assessment of Compliance Assistance Projects." This is an ICR which
was written by EPA to cover a variety of surveys expected to be conducted over the course of
three years on the subject of compliance assistance effectiveness. When OMB approved the ICR,
they agreed to review within 30 days any specific surveys prepared according to the guidelines
within the ICR. The ICR is managed by EPA's Office of Enforcement and Compliance
Assistance (OECA).
As a first step in the process, the ICR itself (Form OMB 83-1 and Supporting Statement)
was obtained, as well as OECA's "Guide for Measuring Compliance Assistance Outcomes"
(EPA300-B-02-011, June 2002) (OECA Guide). Appendix C in the OECA Guide gives a
detailed explanation of how to use the ICR. EC/R and PIRG reviewed this information and then
drafted a survey following the guidelines in the ICR and OECA Guide. The memoranda and
"Request for Approval of Information Collection Activity" that must accompany the survey when
it is sent to OMB were also drafted. The draft documents were then sent to Rochele Kadish and
Lynn Vendinello, the personnel responsible for the ICR at OECA, with whom contact had
already been established. After Ms. Kadish reviewed the draft ICR package, EC/R and PIRG
met with her via conference call to request her comments and feedback on the survey design and
contents. Ms. Kadish was generally very positive about the clarity and appropriateness of the
draft survey questions, in part because they followed OECA's guidance so closely. EC/R and
PIRG then revised the package and submitted the final version to Ms. Kadish for forwarding to
OMB. The email was not recognized as an ICR package to be sent on to OMB and so was not
submitted it in a timely fashion. When this mistake was discovered, the package was submitted
approximately three and a half weeks later than originally planned. When no response was
received within 30 days after the survey was submitted, Ms. Kadish instructed PIRG to assume
the survey had been approved.
21
-------
Several lessons that were learned from the OMB survey approval process and that will be
incorporated into any future compliance assistance evaluation surveys include:
Minimize the number of questions on the survey (and, thus, the response time);
Consider the usefulness of possible responses for each question in the survey;
Establish a rapport with OECA personnel from the beginning of the survey
development process;
Follow the ICR guidance closely for a smooth approval process;
Clearly indicate in your email subject line when you are sending OECA an ICR
package to be forwarded to OMB;
Follow-up on the status of your ICR package; and
If a response to your ICR package for a survey submitted under the Compliance
Assistance ICR is not received from OMB within 30 days, assume it has been
approved.
2. Coordinating with National Technology Services Division
The National Technology Services Division (NTSD) coordinates the tasks required to
post a survey on EPA's public server. These tasks include filing the Application Deployment
Checklist (ADC) and Security Certification, completing staging review of the survey, uploading
the final survey to EPA's public access server, and implementing the collection of statistics on
the number of times the survey is accessed.
PIRG's consultation with the NTSD began after a draft version of the survey had been
developed in Cold Fusion code and the ADC had been filed. After filing the ADC, EC/R and
PIRG met with NTSD to review it and discuss the general process of posting the survey on
EPA's public access server. During the consultation, the NTSD indicated that only applications
developed using Domino code, rather than Cold Fusion, are posted on the server. Since this
survey was to be used as a template for other surveys, Mr. Link decided that the survey should be
re-developed using the approved Domino code. The survey was re-developed using Domino
code with a Lotus 1-2-3 back-end database and appropriate changes were made to the ADC.
After completing a beta-test and making final changes to the survey, both the survey and
a Security Certification were submitted to NTSD. The Security Certification was completed by
PIRG and submitted by Bill Harnett, the director of ITPID. The Security Certification form is
included as Attachment 4 to this report. The NTSD posted the survey on EPA's public access
server, provided with the URL for the survey, and coordinated the collection of statistics to track
the number of hits the survey page received.
22
-------
Several lessons that were learned from working with NTSD and that will be incorporated
into any future compliance assistance evaluation surveys include:
Meet with NTSD prior to survey development to ensure use of the appropriate
programming and data collection software;
Maintain ongoing communication with NTSD to ensure that the appropriate forms are
completed (Application Deployment Checklist and Security Certification) and EPA
protocols are met;
Update NTSD throughout the survey development process to ensure that the project
timeline allows time for NTSD approval; and
Subscribe to the Domino listserv (e-mail: DominolJsers-owner@lists.epa.gov) to stay
informed of software developments and changes.
3. Technical Issues
The online survey was constructed using Lotus Notes database design software Domino
Designer version 5.0.8. While Domino Designer is commonly used to develop databases for use
on Lotus Notes, it also has the capabilities to develop databases, including surveys, for use on
the internet that can be viewed with common web browsers like Internet Explorer or Netscape.
Domino has an interface to capture survey responses in a table, and then the table can easily be
exported to a Lotus 1-2-3 database.
As with any development platform, technical issues were encountered while developing
the survey with Domino Designer. As mentioned, the first version of the database was developed
using Cold Fusion. Because of restrictions with EPA security, it was decided that Domino
Designer should be used instead. It was thought at first that code from Cold Fusion could be
directly translated into Domino, thus decreasing the time needed to re-create the survey in
Domino. However, the two platforms were incompatible, and the survey had to be redesigned
exclusively in Domino. This created avenue for producing a template for future online surveys
in Domino, and it also created a more straightforward method for the programming aspect of the
survey.
Redesigning the survey in Domino caused some of the original features from the Cold
Fusion version to be lost. The format of the survey was user-defined; that is, different survey
questions appeared depending on the answers to previous questions. Since Cold Fusion has very
few compatibility problems with the internet, more formatting and programming options were
available. As mentioned above, Domino Designer is used primarily to develop applications for
Lotus Notes, which is different from producing a web page for the internet. The programming
languages and basic page design can be incompatible with web browsers, especially Netscape.
Other formatting and programming issues arose while developing the survey in Domino.
For example, it was discovered during the beta-test period that the survey did not view correctly
on some screen resolutions because the programming did not translate well from the back-end
23
-------
database to the internet. Problems were also encountered with the way different survey questions
appeared based on user responses. Some questions did not appear at all, and others seemed to
appear at the wrong times. There were some programming issues with the table that captured
survey responses as well; some questions were not programmed to correctly capture the
responses. These programming issues were lessons learned that were corrected during the beta-
test period.
Three technical issues could not be resolved. The first two issues were due to
programming and time restrictions. First, since the survey was user-defined, some questions did
not appear at all for some users. For these questions, the survey should have automatically
entered a value of "NA" for "Not Applicable" in the back-end database. If users did not answer
questions that did apply to them, the response should have automatically defaulted to "NR" for
"No Response" in the database. Programming and time restrictions forced all the answers to
default to "NR."
Second, in Cold Fusion, the survey was originally designed to check for blank answers.
If blanks were found, a popup screen would ask users if they would like to answer the blank
questions. This quality assurance measure was not possible in Domino without major
programming modifications.
Third, the survey was designed to reload itself with different questions based on user
answers. However, in practice, the survey reloaded to different spots on the web page. For
example, when the user answered the third question, the survey might reload and move up to the
first question. The user would then have to scroll down to find question four. Neither EC/R nor
EPA technical support could resolve this issue within the constraints of available resources.
Throughout the production of the survey, EC/R worked closely with EPA technical
support to ease the transition of the survey to an EPA server. Security clearance is required for
each individual wishing to produce a web site through EPA. EC/R already had two staff certified
to produce certain web sites on EPA, but for future survey development, security clearance will
need to be established at the beginning of the process. EPA allowed the EC/R staff to access the
EPA servers through a gateway called SecuRemote, which can be installed on individual
computers and accessed with an EPA-assigned user name and password. Once the software was
installed correctly, there were no technical problems associated with accessing EPA's servers.
It should also be noted that future surveys will need to be quality checked against EPA's
new Domino server, version 6.0. This survey had no compatibility problems with the new
server, but if future surveys are designed with old software, compatibility may become a
problem. EC/R has recently upgraded its software to version 6.5, and would not have problems
developing surveys in the future. EPA technical support was very helpful with putting the survey
on the EPA server, but they could not provide support on other programming issues. If external
technical support is needed in the future, a subscription to a Domino technical support program
would be helpful.
24
-------
Several technical lessons that were learned from developing the online survey and that
will be incorporated into any future compliance assistance evaluation surveys include:
Due to Domino programming incompatibilities with the internet, determine if the
survey will be viewed by the general public on Internet Explorer, or if it can be
viewed on Lotus Notes, or both;
Establish security clearance with EPA as soon as possible;
Determine if the survey will be user-defined or will require more than one web page
so the programming can be dealt with early in the process; and,
If this survey will be used as a template, the unresolved issues described above will
need to be addressed. None of the issues are fatal, but they will require additional
allocation of resources to solve.
4. Communicating with Stakeholders
The first step in communicating with stakeholders was to determine how to reach them,
and then how to encourage them to participate in the survey. It was decided that potential survey
participants would be most likely to respond to an invitation sent by an individual with whom
they had prior interaction. Ms. Ward was involved throughout the development of the survey
and her on-going contact with the aerospace industry was relied on to reach the majority of the
stakeholders. Ms. Ward had maintained an informal database including e-mail addresses for
stakeholders of the industry and she was willing to invite these stakeholders to participate in the
survey. As an attempt to reach additional stakeholders, PIRG contacted the major trade
association for the industry, the Aerospace Industry Association (AIA). Robert Peters, Director
of Environmental Safety and Health for the AIA, was willing to invite his membership to
participate in the survey.
A review of other EPA-affiliated surveys revealed that, in one instance, an incentive had
been used to increase the survey response rate. By participating in a survey for EPA's
Compliance Assistance Clearinghouse, respondents were entered into a drawing for a prize.
PIRG contacted the developers of that survey to determine if a similar incentive could be used for
this survey. The developers of the Compliance Assistance Clearinghouse survey indicated that
they received approval to offer a prize through the Office of the General Counsel (OGC) and that
it was a project-specific process. Based on this information, PIRG decided not to offer a material
incentive for participating in the survey, and instead stressed in the invitation to the stakeholders
the importance of the survey in improving the implementation tools for other EPA standards,
some of which may apply to aerospace facilities.
After determining who should contact the stakeholders, the challenge remained to
determine the appropriate length of time that should be given for responding to the survey. The
goal was to provide enough time that anyone temporarily out of the office or otherwise occupied
would be able to respond to the survey, while realizing that generally people would provide a
response shortly after receiving an invitation or would not respond at all. While a long survey
25
-------
period might give busy respondents the necessary time to take the survey, a shorter survey period
may provide encouragement to take action for those respondents that might otherwise
procrastinate. Taking this into consideration, the official survey period was set at one month.
This would allow time to prepare draft results for presentation at two EPA conferences occurring
during the month following the survey period. Based on the survey responses, PIRG could
decide whether a reminder invitation should be sent to encourage additional survey response.
The survey received a 10 percent response rate (out of the 135 stakeholders) during the eleven
day period after the original invitation was sent. When no additional responses were received for
the ten days following this period, a "reminder" e-mail was sent encouraging any of the
stakeholders who had not responded to do so in the remaining week of the survey period. The
reminder e-mail was sent to all but 14 of the original stakeholder group, and 12 additional
stakeholders who did not receive the original invitation were sent the reminder note. Following
the reminder e-mail, six additional responses were received within two days for a total response
rate of approximately 15 percent.
Several lessons were learned from communications with stakeholders that will be
incorporated into any future compliance assistance evaluation surveys:
Focus the distribution of the survey on people most likely to have used the tools;
As much as possible when developing implementation tools, maintain lists of names
and email addresses of stakeholders, workshop attendees, and tool recipients in order
to maximize the number of people invited to participate in the survey and,
consequently, the number of survey respondents;
Work with individuals involved in the development of the implementation tools to
identify stakeholders;
Contact trade associations to identify stakeholders;
Conduct the survey closer to the date of development of the implementation tools,
rather than waiting several years;
Solicit responses from people who did not use the tools, but were expected to have
used them, to understand why the tools were not used;
Pursue the idea of using a prize as an incentive to participate in the survey;
Be available to answer questions and provide contact information to potential survey
respondents;
Most respondents will provide their response shortly after becoming aware of the
survey;
A reminder note can be an effective means of increasing response to a survey; and
Posting a notice of the survey on the Air Toxics Website may be an effective means
of publicizing the survey since this is how most respondents obtained the tools.
26
-------
VI. CONCLUSIONS AND RECOMMENDATIONS
While only twenty responses to the Aerospace NESHAP implementation tools survey
were received , the information gleaned from those responses and the lessons learned in the
process of developing the survey will be valuable in future efforts to both develop
implementation tools and evaluate their effectiveness. The major findings from the survey
include:
The Aerospace NESHAP implementation tools were used by the target audiences;
The target audiences found the Aerospace NESHAP implementation tools to be
helpful;
Some respondents among the target audiences did not know about the Aerospace
NESHAP implementation tools, but could have been informed through the Air Toxics
Website and e-mail lists;
The Aerospace NESHAP implementation tools are still being used seven years after
they were developed;
The Aerospace NESHAP implementation tools helped affected sources understand
how to submit reports and make process and equipment changes; and
The Aerospace NESHAP implementation tools made a difference in reducing air
pollution.
In the future, tool effectiveness surveys should be incorporated up-front into plans for
developing implementation tools. When developing implementation tools, EPA should
concentrate its efforts on developing a small number of separate tools, especially a Frequently
Asked Questions document and example forms, which are concise, user-friendly, and readily
available through the Air Toxics Website in a timely manner and in a variety of formats. EPA
should also concentrate on improving the visibility of the Air Toxics Website. Future survey
efforts can reasonably be carried out in a six-to-eight month time period if a generic ICR is used
and a good working relationship is established early with personnel in OECA, NTSD, and the
industry in question.
One aspect of this evaluation that needs further examination is the low response rate. A
strategy should be devised for increasing the response rate on future surveys without violating
anyone's privacy or relying too heavily on trade associations. The following are a few
recommendations for improving the response rate:
Advertise that survey respondents can enter to win a prize;
Create a link to the survey from the Air Toxic Website page containing the
implementation tools;
Directly contact EPA Regional staff who have worked with the industry; and
Develop a listserv for people who wanted to stay informed about the implementation
tools and use the listserv to announce the survey.
27
-------
Finally, the various lessons learned through this evaluation study should be incorporated
into a user guide and accompanying electronic survey template to streamline future survey
efforts. This would facilitate including an effectiveness survey as a standard aspect of the
process for developing implementation tools for OAQPS' regulations. The feedback from such
surveys will help PIRG develop even more useful implementation tools.
28
-------
Attachment 1: The Aerospace Implementation Tools Evaluation Survey
-------
SURVEY ON AEROSPACE NESHAP IMPLEMENTATION TOOLS
I Ik' I-.IW is interested ill l^;irniliLi how useful vou found the implementation tools lor the N;ilioii;il
I mission Standards lor I l;i/;ir«.K>ii> Air l*ol 1 ui;iiil> (NISI IAI'i for Aerospace Manufacturing ;iml
Rew ork facilities I Suhpuri (iC i: "Aerospace NISI I Al'" I. I'll cm.* tools can he found ai:
linn: wwu .cna.uov lin alw acrosn acronu.html IMI*. C"oinplclinu ihis survev should lake no more
Ulan live minutes. Your volunlarv inpiu will help I l\\ develop heller implementation tools for
oilier NISI IA IN and respond more effect i \ el_v lo vour needs. All survev s are anon villous: we w ill
nol ohiain or More information from \our computer Mich as vour II1 address, and no "cookies" will
he used.
^ oil do nol need lo complete this sui\e\ if ;tn\ of the follow in<; :i|)|)l> to \oii:
You are nol and hav e nev er heen inv olved in anv w av w iih ihe aerospace indusirv
I'liere are no aerospace facilities in vour jurisdiction
If this is the case, llie I PA would appreciate il if vou would pass the I Rl. I'or lhis survev on lo ihe
most appropriate contact in vour organi/alion.
1 I Willi which l\|>c of organization are vou arriliated'.'
~ AI'lee led source ~ I'rade association ~ federal government
~ State government ~ Local government ~ Consulting firm
~ I.aw firm ~ ( Mlier I please specil'v i:
2a) Did you obtain copies of any of the implementation tools for the Aerospace NESHAP?
~ Yes (go to question 2c) ~ No
2hi Win did vou nol obtain copies of anv of the implementation tools for ihe Aerospace Nl SI IA l*".' n ihl
survey)
~ I didn't know ihev existed ~ I am nol the appropriate person lo use them
~ I tried hut was unahle lo ohtain copies ~ ( Mlier I please specil'v i:
2c) How did you obtain the implementation tools for the Aerospace NESHAP (check all that apply)?
~ EPA's Air Toxics Website ~ EPA's Compliance Assistance Clearinghouse
~ Hard copy direct through the mail ~ At a training session or conference
~ A colleague gave me a copy/link ~ Other (please specify):
2d) When did you obtain the implementation tools lor the Aerospace NESHAP?
~ By the compliance date (9/1/98) ~ Alter the compliance date (9/1/98)
~ I don't remember ~ Some tools by the compliance date & some tools alter
the compliance date
3a) Have you used any of the implementation tools for the Aerospace NESHAP?
~ Yes (go to question 4a) ~ No
Al-2
-------
.Mil Wh\ did \mi not use an\ of the iniplcniciiiiiiinn tooU lor the Acri»piicc Nl Sll.\l>'.' /iW.w/nvn
~ I received them loo hue
~ lhe> \\eiV loo difficult lO understand
~ I \\a> nol the iipprnprialc |V1'm>ii lo have received the looU
~ The lools were not helpful to me
~ ( Mher (please >pecif\ i:
4a) Please rate the usefulness of each of the implementation tools for the Aerospace NESHAP that you
have used. Each tool name shown below is hyperlinked to an electronic copy of the actual tool for
your reference. [For all government/consultant respondents:] Also indicate for which purpose(s)
Tool
I found the usefulness of this tool to be:
I used this tool for the following
purpose(s):
Very
Good
Good
Satis-
factory
Poor
Very
poor
Didn't
use
Compliance
Assistance
Permit-
ting
Enforce-
ment
Overview Brochure
Compliance Timeline
Regulatory Overview
Applicability Flowcharts
Inspection Checklists
Example Recordkeeping
Forms (1,2,3,4,5,6,7)
Example Calculations
Example Reporting
Forms (1,2,3)
Frequently Asked
Questions (Q&A's)
Applicability Guidance
Memos (1,2)
List of Manufacturers of
Method 319 Paint
Arrestors
4h I Are the implementation looU lor the Aerospace NISI IAI* still use III I to \ou nowi. Ill ropmiiLni.t
1W1/J/ Mtlirci'.i git In (///i'.I//'"// 'J)
~ Yes
~ No. the\ are not up-to-date
~ No, they arc not useful for maintaining compliance
~ No. m\ joli no lonuer re^uiro me to u>e them
~ ( Mher I please >pecif\ i:
Al-3
-------
**QUESTIONS 5-8 ARE FOR SOURCES ONLY**
5) The implementation tools for the Aerospace NESHAP increased my understanding of how to
carry out the following regulatory actions (check all that apply):
~ Submitting a notification ~ The implementation tools were not
~ Submitting reports helpful
~ Submitting data ~ Other (please specify):
~ Obtaining a permit
M The implementation tools for the Aerospace NESHAP increased m\ understanding ofliou lo
make the following process and equipment changes (check all that appk):
U (.'hanging the storage or handling of a U Installing pollution control equipment
waste or emission U The implementation tools were not
U (.'hanging a process or practice helpful
U Implementing a material or waste U Other (please specif\):
ivc\cling s\stem
U Installing new process equipment
7) The implementation tools for the Aerospace NESHAP increased my understanding of how to
carry out the following management actions (check all that apply):
~ Instituting training or other ~ Conducting a self-audit
communication to improve awareness ~ The implementation tools were not
and/or practices helpful
~ Establishing an environmental ~ Other (please specify)
management system
Al-4
-------
Sill Do \ou lliink \our u>e ol'llie implementation lool> and \our increased untlcr^UiiitliiijJL ol'llie
Aerospace N1\SI IAP lia> refilled in pollulanl reduction^ al\our Hicilil\
U Ye> U I'm iu)i miiv (>kip lo Nc)
U Noi^kiploNc) U There li:i\e nol been pollulaiil reduction*
<*kip lo Nc)
8b) If available, please provide any details you have on pollutant reductions (e.g., tons per year
[tpy] reduction in specific pollutant). If you would like to provide supporting information,
such as a reference or link to a report or other source of data, you may do so here.
No Do \ou lliink \our u*e ol'llie implementation tool* and \our increased understanding ol'llie
Aerospace Nl-SIIAP lia> resulted in co>l >a\ing>'.'
U Ye* U I'm nol miiv <>kip lo question ^a)
U No (>kip lo question l);i) U There lia\e nol been a»l >a\ ing> (>kip lo
question l);i)
8d) If available, please provide any details you have on cost savings. If you would like to provide
supporting information, such as a reference or a link to a report or other source of data, you
may do so here.
9a) Did you modify any of the implementation tools lor the Aerospace N ESI I AP to suit your needs?
~ Yes ~ No (go to question 10)
9b) How did you modify the implementation tools for the Aerospace NESHAP?
10) How would you improve the implementation tools for the Aerospace NESI IAP?
Al-5
-------
Thank You for participating in the survey
OMB Control No. 2020-0015
Approval expires January 2005
Public reporting for this collection of information is estimated to average 5 minutes per response,
including time for reviewing instructions, gathering information, and completing and reviewing the
information. Send comments on the Agency's need for this information, the accuracy of the provided
burden estimate, and any suggestions reducing the burden, including the use of automated collection
techniques to the Director, Office of Environmental Information, 1200 Pennsylvania Ave., NW
Washington DC 20460 and to the Office of Information and Regulatory Affairs, Office of Management
and Budget, 725 17th St. NW Washington DC 20503. Attention: Desk Officer for EPA. Include the
EPA ICR 1860.01 and the OMB control number 2020-0015 in any correspondence. Do not send your
completed surveys to this address.
Al-6
-------
Attachment 2: Invitation Template for Beta-test Group
-------
SUBJECT: Invitation - Aerospace MACT Implementation Tools Survey
Dear Aerospace Industry Stakeholder:
The U.S. Environmental Protection Agency (EPA) is planning to launch a survey to evaluate how
useful industry stakeholders found the implementation tools that we developed in 1997 for the
National Emission Standards for Hazardous Air Pollutants (NESHAP) for Aerospace
Manufacturing and Rework Facilities. Your input and feedback can greatly benefit the EPA in
developing new implementation tools that are the most useful for stakeholders.The EPA would
appreciate your help in beta-testing the survey before it is distributed.
Please take a few minutes (approximately five) to respond to the draft online survey using the
link below. If desired, you may submit more than one response while testing the survey. After
submitting your response, please provide the EPA with any feedback or suggestions you have,
including indicating any questions or wording that were unclear. In order to incorporate your
feedback in the final version of the survey, your response and any suggestions are needed by
January 23, 2004.
The EPA is interested in distributing the final version of the survey to as many stakeholders as
possible. A distribution list for the final version of the survey is attached to this message in a file
below. If possible, please review the distribution list and provide the EPA with the contact
information (including an e-mail address) for any additional stakeholders. We will also be asking
you to fill out the survey again once it is finalized.
To begin testing the draft Aerospace NESHAP Implementation Tools Survey click on or go to
the following website: http ://www.sscap .net/aerosurvev/
Please submit comments and suggestions to the EPA by sending an EM ATI, to us at:
link.tom@epa.gov
cox. shannon@ecrweb. com
Thank you very much for your help!
Tom
A2-2
-------
Attachment 3: Invitation Template for Stakeholders
-------
Dear Aerospace Industry Stakeholder:
The U.S. Environmental Protection Agency (EPA) is conducting a survey to evaluate how useful
you found the implementation tools that we developed in 1997 for the National Emission
Standards for Hazardous Air Pollutants (NESHAP) for Aerospace Manufacturing and Rework
Facilities. These tools can be found at http://www.epa.gov/ttn/atw/aerosp/aeropg.html#IMP.
Please take a few minutes (approximately five) to participate in the online survey. Your
participation and responses are anonymous and will help the EPA make the most appropriate use
of the resources available for tool development efforts for other NESHAPs (such as Surface
Coating of Miscellaneous Metal Parts and Products and Surface Coating of Plastic Parts and
Products) and residual risk standards. We would appreciate your survey response by March 29,
2004.
The EPA is interested in hearing from as many stakeholders as possible. Please help us in
making this survey a success by forwarding this message to any appropriate individuals and
organizations.
Link to the Aerospace NESHAP Implementation Tools Survey:
http://vosemite.epa. gov/oar/ aerosurvev.nsf/ survey 1 ?openform
Please Note: Depending on the configuration of your computer, you may need to use your scroll
bar to see all of the survey questions. We 're interested in your input, so please be sure not to
miss any questions! If you experience any other significant problems, feel free to call Shannon
Cox at (919)-484-0222 ext. 229.
A3-2
-------
Attachment 4: Security Certification Form
-------
Application Security Certification for Non-Major/Non-Sensitive Web Application
This checklist is only applicable for applications that have not been defined as a "Major Agency System"
according to the IRM Policy Manual Chapter 17, Section 5
http://intranet.epa.gov/rmpolicy/ads/manuals/Chaptrl7.PDF and do not contain or access data/information that
is defined as "High Sensitivity" according to the EPA Information Security Manual 2195A1
http://intranet.epa.gov/rmpolicy/ads/manuals/Manual.PDF. If your application meets the criteria of a "Major
Agency System" or contains data that is "High Sensitivity", you are required to submit an Application Security
Plan http://cfint.rtpnc.epa.gov/ntsdweb/security/SecAppDevChecklist.cfm as set forth in the EPA Information
Security Planning Guidance Document http://intranet.epa.gov/itsecurity/security.pdf
Application Owner Information
Owner Name:
Organization:
Phone Number:
Email Address:
Developer Information
Developer Name:
Company:
Phone Number:
Email Address:
Application Information
Application Name:
Application
Acronym:
ADC Number:
N/A
/
Division Director Certification
I certify that the application is not defined as a "Major Information System".
I certify that the application does not access information that is defined as "High Sensitivity".
I agree that NTSD has the authority to remove the application from production if the application
adversely impacts the Central Production Environment or applications located on the same.
I certify that the application follows all guidelines set forth in the EPA ColdFusion Development and
Deployment Guide http://intranet.epa.gov/coldfusion/devguide/ and ColdFusion Developer Articles
http://intranet.epa.gov/ coldfusion/ articles.html
I certify that the application follows the principles set forth in the EPA Domino WebSite
http://basin.rtpnc.epa.gov/admin/rtpdomino.nsf/ByTitle/StepsToDeploy70penDocument
I certify that all guidelines regarding cgi scripts located at
http://intranet.epa.gov/webmast3/webguide/cgichk.html have been followed.
I certify that the application follows all guidelines regarding Java Server Pages (JSP), Java servlets and
Java beans located at: http://intranet.epa.gov/java/
Division Director:
Email:
@epa.gov
Date:
Note: This document must be Emailed to the ADC Security (Email: adc security) as a WordPerfect attachment
from the Division Director to be considered as proper certification.
A4-2
------- |