.ifiosr,,
Op	So
X ^ OFFICE OF INSPECTOR GENERAL
V PRO^C
Catalyst for Improving the Environment
Evaluation Report
Science to Achieve Results
(STAR) Fellowship Program
Needs to Place Emphasis on
Measuring Results
Report No. 2003-P-00019
September 30, 2003

-------
Report Contributors:
Manju Gupta
Jee Kim
Abbreviations
DOD
Department of Defense
EPA
Environmental Protection Agency
MAI
Minority Academic Institutions
NCER
National Center for Environmental Research
NIEHS
National Institutes of Environmental Health Science
NRC
National Research Council
NSF
National Science Foundation
OIG
Office of Inspector General
OMB
Office of Management and Budget
ORD
Office of Research and Development
S&E
Science and Engineering
STAR
Science to Achieve Results

-------

|	1	UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
o \\7// 
-------
Action Required
In accordance with EPA Manual 2750, you are required to provide this office with a written
response within 90 days of the final report date. The response should address all
recommendations. For corrective actions planned but not completed by the response date, please
describe the actions that are ongoing and provide a timetable for completion.
We have no objection to the further release of this report to the public. Should you or your staff
have any questions, please contact me at (202) 566-0831 or Manju Gupta, Project Manager, at
(202) 566-2478.
For your convenience, this report will be available at http://www.epa.gov/oig/
Attachment
-------
Executive Summary
The Environmental Protection Agency (EPA) established the Science to Achieve
Results (STAR) Fellowship Program to provide funds to graduate students to
encourage them to obtain advanced degrees and pursue careers in
environmentally related fields. The program is administered by EPA's National
Center for Environmental Research (NCER). To determine the overall
effectiveness of the program, we sought to identify specific performance
measures, performance data, and internal evaluations established by NCER to
assess the results of the fellowship program. We attempted to analyze the
demographic composition of the students applying and entering the program.
Additionally, we identified similar fellowship programs at other Federal agencies
and compared approaches to assessing program effectiveness and tracking
applicant and recipient demographic characteristics.
Results in Brief
Program Performance. NCER did not place emphasis on determining the
results and achievements of its STAR Fellowship Program. NCER primarily
focused on selecting the fellows and awarding the grants, without determining
whether the program's goals were being achieved. NCER did not establish
relevant performance measures, nor collect performance data on the
accomplishments of the STAR Fellowship Program. Consequently, the success
of the program in having recipients pursue environmentally related fields cannot
be measured.
Program Demographic Data. NCER did not place emphasis on collecting and
analyzing demographic data in administering the STAR Fellowship Program.
NCER's main focus has been on selecting top quality candidates. Since 1999 the
Minority Academic Institutions (MAI) Fellowships have been part of the STAR
Fellowship Program. There is outreach to the MAIs, and NCER considers the
MAI fellowships to address the need to attract minority applicants and fellows.
However, NCER has not consistently collected nor analyzed the demographic
data of the STAR/MAI applicants nor fellows. Without adequate demographic
data, it is not possible to determine whether the program has been successful in
attracting and selecting a diverse pool of fellows.
Other Fellowship Programs. Review of similar fellowship programs at three
other Federal agencies disclosed that performance of fellowship programs can be
measured and tracked and that diversity among applicants and recipients is
routinely monitored.
i
-------
Recommendations
We recommend that the EPA Office of Research and Development (ORD) direct
NCER to expand the focus of the STAR Fellowship Program regarding
measuring results. This should include conducting internal reviews, selecting
meaningful performance measures, and maintaining necessary data on fellowship
applicants and recipients. We also recommend that ORD direct NCER to collect,
maintain, and review the demographic composition of the STAR applicant pool
and the fellows selected, and adjust outreach efforts accordingly. Further, we
recommend adoption of certain successful procedures used by other Federal
agency fellowship programs.
Agency Comments and OIG Response
In its response to our draft report, the Agency agreed to our recommendations
related to establishing Performance Measures and the need for data collection
and analyses, tracking the fellows after completion of the program, and some of
the procedures used by other agencies. EPA prepared an action plan, including
action officials and due dates for each recommendation. However, EPA did not
agree with our comments on diversity and need for outreach to minorities outside
of the MAI program. EPA suggested we remove the chapter on diversity and
include the recommendation for collecting demographic data with other
recommendations in Chapter 2. We have made some modifications to the report
in response to EPA's request, but did not remove Chapter 3 on diversity and the
collection of demographic data. EPA's comments are in Appendix D, and our
evaluation of those comments is in Appendix E.
-------
Table of C
Executive Summary 	 i
Chapters
1	Introduction 		1
2	Emphasis Needs to Be Placed on Measuring Results of Program		5
3	Demographic Information Not Systematically Collected		9
4	How Similar Federal Agency Fellowship Programs Evaluate
Effectiveness and Monitor Diversity		15
Appendices
A Input for Logic Model Depicting STAR Fellowship Program		21
B Details on Scope and Methodology		25
C Additional Information on Other Fellowship Programs		27
D Agency Comments on OIG Draft Report 		29
E OIG Response to Agency Comments on Draft Report		33
F Distribution		37
Figures
1-1:	Logic Model Depicting STAR Fellowship Program Design 		3
2-1:	STAR Fellowship Program - Summary of File Review		6
3-1:	STAR Fellowship Applicants by Race/Ethnicity		11
3-2:	STAR Fellowship Recipients by Race/Ethnicity		11
3-3:	STAR Fellowship Applicants by Gender 		12
3-4:	STAR Fellowship Recipients by Gender		12
3-5:	MAI Graduate Fellowship Applicants by Race/Ethnicity 		13
3-6:	MAI Graduate Fellowship Recipients by Race/Ethnicity 		13
Tables
3-1:	Profile of STAR Fellowship Program Applicants/Recipients by Race/Ethnicity ...	11
3-2:	Profile of STAR Fellowship Program Applicants/Recipients by Gender 		12
3-3:	Profile of MAI Graduate Fellowship Applicants/Recipients by Race/Ethnicity ....	13
4-1:	Goals for Other Agency Programs		16
4-2:	Evaluation/Tracking by Other Agency Programs		17
4-3:	Diversity Efforts for Other Agency Programs		18
C-1:	Selection Processes for Other Agency Programs	27
C-2:	Project Officer Duties for Other Agency Programs 	28
-------
-------
Chapter 1
Introduction
Purpose
The objective of this evaluation was to determine whether the Science to Achieve
Results (STAR) Fellowship Program established by the Environmental Protection
Agency (EPA) Office of Research and Development (ORD) has been effective in
encouraging promising students to obtain advanced degrees and pursue careers in
environmentally related fields, and to determine the demographic profile of the
applicants and fellows, who are selected to become the future generation of the
environmental work force. Specifically, we sought to:
Identify the specific performance measures used by ORD to evaluate the
STAR Fellowship Program.
Identify the demographic profile of the applicant pool, and compare it with the
demographics of the fellowship recipients.
Compare the STAR Fellowship Program with similar fellowship programs at
other Federal agencies.
Background
EPA established the STAR Research Grants Program in 1995, in response to
specific needs identified by Congress. The STAR program, administered by
ORD's National Center for Environmental Research (NCER), is made up of three
components: (1) Focused Requests for Applications; (2) Exploratory Research;
and (3) STAR Graduate Fellowships, including the Minority Academic
Institutions (MAI) Fellowships. This report only addresses the third component.
STAR Fellowship Program
The STAR Fellowship Program provides funds to graduate students in
environmentally related fields at colleges and universities across the country.
Fellowships are awarded to competitively selected students for Master's and
Doctorate programs in physical sciences, biological sciences, and engineering.
NCER describes the objective of the STAR Fellowship Program as:
To encourage promising students to obtain advanced
degrees and pursue careers in environmentally related
fields. . . . This program will benefit both the public and
private sectors which will need a steady stream of well-
trained environmental specialists if our society is to meet
the environmental challenges of the future.
1
-------
Approximately $10 million of the STAR Grants Program has been devoted
annually to the STAR Fellowship Program. EPA has spent approximately
$69 million to fund over 800 STAR and MAI Fellows at 168 United States
colleges and universities from 1995 through 2001. NCER did not put out Request
For Applications for STAR Fellowship awards for fiscal 2002 because funding for
the program was not included in the EPA budget. However, funding of
$9.75 million was restored for the fiscal 2003 budget year. There is continuing
uncertainty about the level of funding for the fellowship program in the future.
Minority Academic Institutions Fellowships
ORD established the MAI Graduate/Undergraduate Student Fellowship Programs
in response to Executive Order 12320 (September 15, 1981), "to strengthen the
capacity of historically black colleges and universities to provide excellence in
education." Based on White House Initiatives, the list of participating minority
institutions grew to include Hispanic Serving Institutions, Tribal Colleges,
Alaskan Native Serving Institutions, and Native Hawaiian Serving Institutions.
The MAI fellowships were administered under the National Consortium for
Graduate Degrees (GEM program) until 1998, at which time NCER took the
programs over as part of the STAR Fellowship Program. The goal of the MAI
programs is to assist eligible minority academic institutions in providing
education and training to fellows interested in pursuing environmental careers and
to strengthen environmental research programs at these institutions. An
applicant's eligibility is based on attending a minority academic institution,
without consideration being given to the applicant's race or ethnicity.
National Research Council Report
The National Research Council (NRC) of the National Academies of Science in
April 2003 completed an independent assessment, sponsored by ORD, of the
STAR Research Grants Program.1 The NRC established a committee for the
review made up of representatives from universities, environmental groups, the
American Chemistry Council, and a State environmental protection agency. The
committee was charged to assess the program and "recommend ways to enhance
the program's scientific merit, impact, and other benefits." The report describes
the STAR Fellowship Program as "a small component of the overall STAR
program whose goals and objectives differ from those of the main research grants
program." In evaluating the fellowship program, the committee found, "although
the program publishes on the NCER website information about all the students
receiving fellowships, it does not gather systematic information to track the status
of past and currently funded fellows to assess the impact of the STAR program on
their careers. To gather information on the influence of the fellowship program,
1 The Measure of STAR Review of U.S. Environmental Protection Agency's Science to Achieve Results
(STAR) Research Grants Program, National Research Council of the National Academies, April 2003.
2
-------
the committee contacted more than 100 STAR fellows who were initially funded
in 1995 and 1996 and who would have completed their graduate work." The
NRC reported that nearly 90% of the former fellowship recipients they contacted
remained in the environmental field. The committee's report concluded, "Given
the nation's continuing need for highly qualified scientists and engineers in
environmental research and management, the STAR fellowship should be
continued and funded."
Logic Model Depicting Design of STAR Fellowship Program
Figure 1-1 represents a logic model that illustrates the general design of the STAR
Fellowship Program, and how it is intended to support EPA and ORD.
Figure 1-1: Logic Model Depicting STAR Fellowship Program Design
Performance Measures
Resources
EPA/NCER
FTEs -Managers
-Project Officers
Peer Review
Panels
Funding -
$10 million/year
1995
Appropriations
Act
Universities
-facilities
-faculty
National Science
Foundation
Grants
Adm inistration
Division
Activities
Outreach
* Selection
Process
-Draft requests
for applications
-Select peer
review panels
-Relevance
Funding and
Renewal Process
Plan & Arrange
Conference
Closeout and
Tracking
-Collect thesis/
dissertations
-tracking after
completion
Outputs
Awards for
Environm ental
~ Studies and
Research
-Large applicant
pool of brightest
students
-Request for
applications
Progress Data on
Fellows
Term ination
notices
Fellows
Conference
Post-completion
Data
Database on post
fellowship
activities
Customers
Students
Universities,
Colleges,
Research
Institutions
EPA Program
Offices
Employers
-Federal
-Industry
-State and local
govemm ents
-Non-
govemm ent
organizations
Short-Term
Outcome
Increased
number of
fellows with
Masters and
Ph.D.
degrees go into
env ironm ental
fields for
continuing
education /
research / or
jobs/careers
Steady stream of
highly educated
and technically
trained
env ironm ental
professionals
available for
national
workforce
Researchers
work on science
issues related to
env ironm ent
Intermediate
Outcome
Nation has a
steady stream of
well trained
environmental
specialists who
-develop sound
science
-	solve
environmental
issues
-	make better
policy decisions
Long-T erm
Outcome
Improved human
health
Improved
environm ental
quality
Improved
ecosystem health
Externalities
Congress, OMB, Funding, other Federal agency programs, universities/academic institutions, national economic climate
3
-------
The logic model places the fellowship program in the larger context of ORD and
EPA goals, and the conceptual links between the goals of fellowship program and
ORD's and EPA's goals. It shows what the program does, the resources used for
its activities, and the customers it is designed to serve with its outputs. The model
illustrates the links between design, implementation, and outcomes, and contrasts
design outcomes and program manager priorities. Details on the content of the
logic model are in Appendix A.
The design of the STAR and MAI fellowships are similar, with minor differences.
Therefore, we are only presenting the STAR logic model, and the differences are
discussed in Appendix A as appropriate.
Scope and Methodology
We began our review on September 19, 2002, and completed field work on
April 25, 2003. We conducted our work in Washington, DC. We performed our
review in accordance with Government Auditing Standards, issued by the
Comptroller General of the United States.
In order to determine the degree to which the objectives of the STAR (including
MAI) Fellowship Program were met, and how it contributes toward ORD and
EPA goals, we reviewed the design of the program for logic and completeness by
preparing a logic model. We obtained information through web sites and program
files, reviewed the NRC report, and conducted interviews with ORD staff. We
shared a draft of the logic model and the explanatory notes (Appendix A) with
NCER managers, and received their comments, which we have incorporated as
necessary. We selected a random sample of fellowship files for review. The
results of the review of files are in Chapters 2 and 3. We also reviewed the
fellowship programs for three other Federal agencies (Department of Defense,
National Science Foundation, and National Institutes of Environmental Health
Sciences) for comparison and documenting their procedures (Chapter 4).
We have reported the data on demographics as provided by NCER staff in
Chapter 3, but did not independently test the accuracy of the data. Our analyses
were limited because NCER has not collected data systematically on
demographics, or on jobs/careers/activities after completion of fellowships.
Regarding prior coverage, we reviewed prior EPA Office of Inspector General
(OIG) Report No. 2002-P-000002, "Design for Objective 8.4 Could Be Improved
by Reorienting Focus on Outcomes," dated November 21, 2001.
Additional details on our scope and methodology are in Appendix B.
4
-------
Chapter 2
Emphasis Needs to Be Placed on Measuring
Results of Program
NCER did not place emphasis on determining the results and achievements of its
STAR Fellowship Program. NCER primarily focused on selecting the fellows
and awarding the grants, without determining whether the program's goals were
being achieved. Consequently, the success of the program in having recipients
pursue careers in environmentally related fields cannot be measured and evaluated
because NCER did not collect sufficient performance measurement data. This is
consistent with the observation in a prior report2 that "ORD focuses primarily on
outputs. Placing greater focus on outcomes rather than the steps taken to achieve
outcomes could result in more valuable benefits being realized."
Sufficient Data Not Collected
Based on our review of a sample of 50 randomly selected files for STAR
Fellowship Program recipients from 1995 through 1999 (10 folders for each year),
we concluded that NCER did not collect sufficient information to determine
whether fellows obtained an environmentally related degree or pursued an
environmentally related career. Most program activities dealt with selecting the
fellows and dispensing resources.
Specifically, we found that NCER did not track the fellows after termination of
their fellowship. This was despite the fact that fellowship terms and conditions
require fellows to notify the EPA project officer of all employment for at least
3 years after graduation. When fellows fail to report, EPA does not have a
procedure in place to collect the data. In our initial meetings with NCER staff, we
were told that NCER planned to track fellowship recipients' career choices and
professional activities as related to the environment for 5 years. However, when
we searched the folders to check how much information was collected on the
employment or professional activities of the fellows after 3 years, we did not find
any information in the files for completed fellows on post completion activities,
such as career/jobs.
Information on degree completion, thesis completion, and publications could be
used as interim or partial measures of fellowship success, and we sought to
determine whether NCER was collecting this type of data as well. We found
2Design for Objective 8.4 Could be Improved by Reorienting Focus on Outcomes, OIG Report No.
2002-P-000002, November 21, 2001.
5
-------
20 out of 50 files (40 percent) included information on degree completion, and
29 out of 50 files (58 percent) included completed theses or papers published.
Figure 2-1 provides a breakdown of the type of information collected by NCER
included in the files. This includes progress reports for the first and second years
(Progress Reports #1 and #2, respectively), termination notices,3 an indication of
degree completed, whether a thesis or paper was published, post-fellowship
information in general, and whether career information for the first 3 years after
fellowship termination was included.
Figure 2-1: STAR Fellowship Program - Summary of File Review
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Progress Progress Termination Degree Thesis/Papers Post Career Info
Report #1 Report #2 Notice Completed Published Fellowship (after 3 years)
Info
File Search Elements
(% available in files 1995-1999)
Lack of Evaluation System and Performance Measures
NCER had not established a system of internal evaluations to assess the
performance of the fellowship program. NCER management stated that no
internal evaluations of fellowship programs had been conducted. However, they
view the statistics on the number of people applying each year and the competitive
acceptance rate (approximately 10 percent of applicants receive fellowships) as
indicators of program success. As noted earlier, this does not indicate the number
of recipients who pursue careers related to the environment, which is the ultimate
goal of the program.
3 Details in Appendix A, in "Close-out and Tracking" section.
6
-------
Also, NCER had not established performance measures necessary for monitoring
and reporting the program's accomplishments. A key aspect of results-based
management is setting meaningful performance measures. By way of
performance measurement, NCER only reviews the number and quality of
applicants. However, the number and quality of those selected for awards only
indicate what goes into the program; they do not measure the results of the
fellowship program or the difference the program makes in creating the next
generation of environmental specialists. To determine that information, such
performance measures as degree completion rates, jobs and careers pursued after
fellowship, publications, and academic appointments should be tracked and
assessed over time.
NCER did not collect relevant data necessary for performance measurement and
analyses, as well as for making necessary changes for improvement. It is not
possible to determine how well a program is working or what it is achieving
without collecting data dealing with the stated program goals, which in this case
were students obtaining advanced degrees and pursuing careers related to
environmentally related fields.
NRC Study Noted Similar Issues
A study coordinated by the NRC at the request of ORD (see Chapter 1) noted
similar issues. The NRC report noted that the STAR Fellowship Program does
not gather systematic information to track the status of past and currently funded
fellows to assess the impact of the STAR program on their careers. While the
study noted the selection process ensured selection of high quality applicants, it
also noted NCER did not collect information on degree completion rate or on how
many of the fellowship recipients who completed graduate work went on to work
in environmental sciences. The NRC study recommended that EPA may want to
collect information on publications and other products of fellowship recipients to
document the success of the program.
Recommendations
We recommend that the Assistant Administrator of ORD direct NCER to take the
following actions to effectively measure for results:
2-1. Conduct Internal Evaluations: Periodic in-house evaluations of the
program need to be performed by management to determine whether
resources are being allocated to yield successful outcomes, and to identify
areas that need improvement. Regular evaluations performed in-house and
by external panels are important parts of a program design. Results of
these evaluations will provide the basis for making appropriate changes to
strategies, objectives, or areas of emphasis.
7
-------
2-2. Establish Performance Measures: NCER needs to develop performance
measures appropriate for measuring the outcome of the fellowship
program. The measures should be cost effective, and aligned with EPA's
and ORD's environmental goals. They should provide accurate and timely
information for providing a coherent and clear picture of what the
resources spent are producing short- and long-term. Based on review of
other studies of fellowship programs, some traditional measures for
evaluating fellowship programs include:
a.	Percentage of fellowship recipients who obtained advanced degrees.
b.	Number of fellowship recipients who have completed graduate work
and are working in environmental fields.
c.	Number of fellowship recipients who have completed degree programs
with at least one peer-reviewed publication, or have attained career
placements in research programs or acquired tenure track positions.
d.	Professional productivity in the form of scholarly publications,
professional presentations, and procurement of research grants.
2-3. Collect Data: Data on fellows completing the program must be collected
regularly and analyzed, in order to properly assess the effectiveness of the
program. The availability of adequate data serves as the cornerstone for
the above-mentioned recommendations, since the availability of data will
allow appropriate outcome-oriented performance measures and
evaluations to be applied to the fellowship program.
Agency Comments and OIG Response
ORD agreed with our recommendations to conduct internal evaluations, establish
performance measures, and collect data. However, while ORD indicated that the
appropriate data is currently being collected, we generally did not find that to be
the case, and ORD needs to take further action in this area. ORD's comments are
in Appendix D, and our evaluation of those comments is in Appendix E.
8
-------
Chapter 3
Demographic Information
Not Systematically Collected
NCER has not placed emphasis on evaluating the demographic composition of the
applicant pool, and of the fellows selected, who will be trained to become the next
generation of environmental professionals through STAR fellowships. Even
though having a diverse pool of applicants and fellows is not an objective of the
STAR Fellowship Program, inherent in NCER's objective of meeting the need for
a steady stream of well-trained environmental specialists is EPA's commitment to
a diverse work force that includes participation of women, under-represented
minorities, and people with disabilities. NCER considers the MAI program to be
its response to the need to address diversity. However, the purpose behind the
MAI program under a White House Initiative is to assist in expanding minority
academic institutions' capacity to provide quality education. The eligibility for a
MAI fellowship is attending an institution that has been recognized as an
historically minority academic institution, not the student belonging to a minority
group. Moreover, since NCER did not collect demographic information
consistently, it was not possible to provide a demographic profile of the program.
By the same token, it is not possible for NCER to objectively demonstrate
whether the STAR and MAI efforts are successful in attracting women and
traditionally under-represented minorities.
Need for Increasing Diversity in Science and Engineering Workforce
There have been many recent reports on the business need for increasing
diversity in the science and engineering workforce. We noted two such recent
studies that have emphasized the need to tap into talent from groups that are
under-represented in the Science and Engineering (S&E) workforce:
One study4 notes that: "The demographics of the United States are changing.
Women and minorities make up 60 percent of the total workforce, but they
are dramatically under-represented in S&E. Women comprise 46 percent of
the workforce but only 23 percent of the S&E labor force. African
Americans and other ethnic minorities constitute 24 percent of the total
population but only 7 percent of the S&E labor force. This means the
majority of Americans are under-represented in S&E." The study also notes
4 Envisioning a 21st Century Science and Engineering Workforce For the United States: Tasks for
University, Industry and Government, Report to the Government-University-Industry Research Round Table,
National Academy Press, ISBN 0-309-08856-9 (2003).
9
-------
the lack of adequate and reliable data that could be applied for policy
making purposes.
A second report5 was issued from a forum organized by the National
Academy of Engineers Committee on Diversity in the Engineering
Workforce, with representatives of corporations that have been recognized
for their successful diversity programs. The first and foremost concern
expressed in the study was the need for talented workers and the difficulty of
finding enough qualified personnel. Other issues were the competitive
advantage of having a diverse workforce, and the recognition that engineers
from different ethnic, gender, and cultural backgrounds bring unique and
valuable life experiences to the workplace and encourage creative
approaches to problem solving and design.
In our review of other fellowship programs, we found that increasing diversity in
the future generation of workforce is included as one of the objectives of some
fellowship programs. The data are collected, analyzed, and used in making
changes in the outreach if necessary (Chapter 4).
Collecting and Monitoring Demographic Data
Important for Assessing Successful Outreach
NCER did not consistently collect or analyze demographic data for applicants,
nor the fellows selected, since increasing diversity in the applicants or fellows is
not a stated objective of the STAR Fellowship Program. The main emphasis of
the program is on selecting highly motivated and talented students interested in
environmental fields, from different geographical regions. In response to our
questions on diversity and demographic data, NCER managers told us they
address the issue of diversity through the MAI fellowships. However, the
eligibility for a MAI fellowship is based on attending a MAI institution, rather
than the ethnicity or gender of the applicant or fellow. The goal of the MAI
program, in place since 1981 as a result of a White House Initiative, is to
strengthen the capacity of MAIs - institutions that have historically been attended
predominantly by minority students - to provide excellence in education. A
highly competitive minority candidate who is not enrolled in a recognized MAI
does not meet the eligibility criteria for a MAI Fellowship.
Moreover, NCER did not collect or maintain a database on the demographics of
the STAR and MAI program applicants, nor of those receiving fellows, in 1995-
1998. In 1999, NCER established a database that includes information on the
gender and ethnicity of applicants and fellows. Even after that, the information
was not complete, as shown in following charts. According to NCER managers,
diversity in Engineering: Managing the Workforce of the Future, National Academy Press (2002).
10
-------
the reason for incomplete information is because it is provided by applicants on a
voluntary basis. Moreover, NCER has not placed emphasis on maintaining the
database on diversity in the STAR Fellowship Program. As seen in the following
charts, there are gaps in the recipient data. The fellows are in contact with NCER
staff, thus the data can be collected from them to complete the database.
Demographic data on the number of fellowship applicants and those actually
receiving a fellowship is provided by race (Table 3-1 and Figures 3-1 and 3-2)
and by gender (Table 3-2 and Figures 3-3 and 3-4). As demonstrated by the
"Blank" columns in the tables, demographic information on race and gender was
not available for more than half of the STAR Fellowship Program recipients.
Therefore, the information is of limited usefulness regarding trends, or to assess
whether any changes need to be made to increase minority applicants. Further,
NCER has not collected data on the participation of applicants or fellows by
people with disabilities in the fellowship program. As discussed further in
Chapter 4, other agencies place greater emphasis on collecting and analyzing
demographic data.
Table 3-1: Profile of STAR Fellowship Program Applicants/Recipients by Race/Ethnicity
RACE
White
African Am.
Hispanic
Asian
Native Am.
Blank
Total
Year*
App.
Rec.
App.
Rec.
App.
Rec.
App.
Rec.
App.
Rec.
App
Rec.
App.
Rec.
2000
546
17
33
1
27
1
28
1
9
0
653
87
1,296
107
2001
924
64
40
1
50
3
46
4
5
0
332
28
1,397
100
Total
1,470
81
73
2
77
4
74
5
14
0
985
115
2,693
207
App.: Applicants
Rec.: Recipients
* Funding to provide new fellowships was not provided in 2002.
Figure 3-1: STAR Fellowship Applicants by Race/Ethnicity
White - 54.6%
African Am.- 2.7%
Hispanic-2.9%
Asian - 2.71
Native Am. - 0.
Blank - 36.6%
Figure 3-2: STAR Fellowship Recipients by Race/Ethnicity
White-39.1%
African Am. -1.0%
Hispanic -1.9%
Asian - 2.4%
Blank - 55.6%
11
-------
Table 3-2: Profile of STAR Fellowship Program Applicants/Recipients by Gender
GENDER
Male
Female
Blank
Total
Year*
App.
Rec.
App.
Rec.
App.
Rec. **
App.
Rec.
2000
583
11
698
17
15
79
1,296
107
2001
550
33
746
41
101
26
1,397
100
Total
1,133
44
1,444
58
116
105
2,693
207
App.: Applicants
Rec.: Recipients
* Funding to provide new fellowships was not provided in 2002.
** The numbers were provided by NCER. The greater number of blanks for recipients compared to applicants
shows the database was not updated for recipient information.
Figure 3-4: STAR Fellowship Recipients by Gender
Figure 3-3: STAR Fellowship Applicants by Gender
Male-42.1%
Female - 28.0%
12
-------
Table 3-3 provides demographics for the MAI graduate fellowship program
applicants and recipients, and Figures 3-5 and 3-6 provide illustrations. It should
be noted that an applicant's eligibility for an MAI fellowship is based on an
applicant's attending a minority academic institution rather than the applicant's
race or ethnicity.
Table 3-3: Profile of MAI Graduate Fellowship Applicants/Recipients by Race/Ethnicity
RACE
White
African Am.
Hispanic
Asian
Native Am.
Blank
Total

Year
App.
Rec.
App.
Rec.
App.
Rec.
App.
Rec.
App.
Rec.
App
Rec.
App.
Rec.

2000
3
0
28
2
15
1
1
0
5
0
11
5
63
8

2001
8
0
27
5
26
9
2
0
0
0
11
5
74
19

2002
24
9
28
2
23
2
6
1
0
0
14
6
95
20

Total
35
9
83
9
64
12
9
1
5
0
36
16
232
47

App.: Applicants
Rec.: Recipients
Figure 3-5:
MAI Graduate Fellowship Applicants by Race/Ethnicity
African Am. - 35.8%
Hispanic - 27.6%
White-15.1%
Blank-15.5%
Native Am. -2.2%
Asian - 3.9%
Figure 3-6:
MAI Graduate Fellowship Recipients by Race/Ethnicity
African Am. -19.1%
Hispanic - 25.5%
White-19.1%
Blank - 34.0%
13
-------
Recommendation
We recommend that the Assistant Administrator of ORD direct NCER to take
the following action regarding diversity:
3-1. Collect and Analyze Demographic Data. NCER needs to systematically
collect and analyze data on relevant demographic information (race,
gender, persons with disabilities) of applicants and incoming fellows, and
adjust outreach efforts as needed to increase the pool of applicants of
certain demographic subsets that may be under-represented in the
fellowship program.
Agency Comments and OIG Response
ORD did not agree with our comments on diversity and the need for
additional outreach to minorities outside of those attending MAIs, and
suggested we remove the chapter from the report and include our
recommendation on collecting demographic data in Chapter 2. We have not
removed the chapter, but have made some modifications to make it clearer
that our main concern was that ORD did not have sufficient demographic
data regarding diversity. ORD's comments are in Appendix D, and our
evaluation of those comments are in Appendix E.
14
-------
Chapter 4
How Similar Federal Agency Fellowship Programs
Evaluate Effectiveness and Monitor Diversity
We compared EPA's STAR Fellowship Program with similar fellowship
programs at other Federal agencies, and noted some procedures at those other
agencies on how to meet and evaluate goals that EPA might be able to apply.
While the STAR Fellowship Program has been in existence since 1995, other
fellowship programs have existed for much longer. To compare the design,
execution, and assessment of comparable programs, we met and consulted with
program coordinators of fellowship programs at three other agencies to discuss
their programs and gathered information on procedures that could be applied to
EPA. Those three other agencies, as well as the particular program we reviewed
for each, were:
Department of Defense (DOD):
National Defense and Science Engineering Graduate Fellowship
National Science Foundation (NSF):
Graduate Research Fellowship
National Institutes of Environmental Health Science (NIEHS),
part of the National Institutes of Health:
National Research Service Awards Institutional Training Grants
To present a comprehensive picture of the other fellowship programs, we are
presenting information on the organization/management of fellowship programs,
diversity, and evaluation/tracking. We collected, but did not independently
verify, the information provided by program representatives. Information on the
selection process and project officers can be found in Appendix C.
Organization/Management of Fellowship Programs
Goals
In line with the goal established for EPA's STAR Fellowship Program to
"encourage promising students to obtain advanced degrees and pursue careers in
environmentally related fields", all three fellowship programs have broadly stated
goals for their programs that aim to help increase and/or produce the next
generation of scientists and engineers by providing selected fellows with funding
for graduate-level studies at academic institutions across America. The goals for
the three other programs are in Table 4-1.
15
-------
Table 4-1: Goals for Other Agency Programs
DOD
NSF
NIEHS
Increasing the number and
quality of our Nation's scientists
and engineers."
"Aims to ensure the vitality of	"To support research training
the human resource base of	opportunities for individuals . . .
science, mathematics and	who are training for careers in
engineering in the United States	specified areas . . . pertinent to
and to reinforce its diversity."	the environmental health
sciences."
Evaluation and Tracking
All the fellowship programs have tracking systems that depend heavily on the
voluntary participation of fellows and academic institutions. They have
developed procedures to collect and analyze information to evaluate the
performance of their program. They make adjustments and improvements to
their objectives, outreach, data collection, and resource allocation based on the
results of their analyses.
DOD tracks data on diversity of applicants and fellows selected annually. NSF
has maintained a detailed database with demographic data of all applicants and
fellows from the beginning of its Graduate Research Program, established in
1952.6 Further, NSF participates in an annual Survey of Earned Doctorates,
which can be used to track completion rates and time to degree of fellows by
gender, ethnicity, and discipline. NSF sponsors a large-scale evaluation of its
fellowship program by an external entity once every 10 years to objectively
evaluate how effectively the program is achieving its goals. NSF redefines its
objectives according to the outcome of the study. As part of evaluating the
institutions for continued funding, NIEHS requires the academic institutions to
track the students for 10 years after completion of fellowship, and if this
requirement is violated the institution is denied continued funding. Details are in
Table 4-2.
6 National Science Foundation Graduate Research Fellowship Program Final Evaluation Report, WestEd,
September 2002.
16
-------
Table 4-2: Evaluation/Tracking by Other Agency Programs
DOD
NSF
NIEHS
•	The closeout package
contains a survey that asks for
contact information, date of
completion, dissertations /
publication title(s), name of
academic advisor, name of
doctoral institution, current
employer, current position,
information on other Federal
research money received, and
feedback on the program
•	75 percent response rate to
the survey
•	Survey compiled into a report
every year
•	Has maintained a database of
fellows since the beginning of
the program in 1952
•	Tracks completion rates at the
doctorate level annually
•	Sponsors an extensive
evaluation of the fellowship
program once every 10 years
to objectively evaluate how
effectively the program is
achieving its goals
•	To receive continued funding,
academic institutions are
required to track students for
10 years after completion
•	Tracks students that later
become National Institutes of
Health grantees
•	Piloted a web-based system
for tracking as a response to
Office of Management and
Budget's paperwork reduction
requirement
Diversity
The fellowship programs reviewed in this section all use different approaches in
their efforts to increase diversity in their respective programs. DOD does not
incorporate diversity into the selection process, basing selection solely on
academic merit. Nonetheless, they collect the data and report on the
demographics of the applicants and fellows to headquarters. NSF makes efforts
to increase the presence of under-represented minority groups in the applicant
pool by focusing outreach and recruiting toward professional minority
organizations and attracting best quality minority students from academic
institutions. As part of the National Institutes of Health, NIEHS requires
academic institutions to have a plan and a report on recruitment and retainment
of minorities in place at the time of application. NIEHS also has a separate
minority fellowship program. EPA's STAR Fellowship Program does not
incorporate diversity in the selection process, but has the MAI Programs. Details
are in Table 4-3.
17
-------
Table 4-3: Diversity Efforts for Other Agency Programs
DOD
NSF
NIEHS
•	Diversity is not considered in
selection process, selection is
strictly based on merit
•	Drafts a report documenting
the number of minorities who
applied and are selected; also
documents information on
gender figures
•	Has been more active in
focusing outreach efforts
toward professional groups
that cater to under-
represented minorities and
minority academic institutions
to increase presence of
minorities in applicant pool
•	Looking at encouraging
minorities attending top
research schools to apply
to program
•	There is a requirement in the
grant application that
stipulates all institutions need
to have a written plan for
recruitment and retention of
minorities (statistics
demonstrating
implementation of the plan
are evaluated by peer
reviewers)
•	Also has a separate program
of individual fellowships for
under-represented minority
doctoral candidates
Recommendations
Based on our review of other fellowship programs and a comparison against the
STAR Fellowship Program, we recommend that the Assistant Administrator of
ORD direct NCER to take the following actions to improve the STAR
Fellowship Program in relation to evaluation and tracking:
4-1. Include a Post-Fellowship Survey: In line with the practice utilized by
DOD, NCER could include a survey-type form to the termination package.
A survey that asks the fellows to provide important post-fellowship
information could increase the response rate of fellows considerably, as
opposed to allowing fellows to voluntarily submit information. The
termination notice currently used by NCER asks only for termination date,
expected date of degree completion, and "additional comments." The
survey, in addition to asking for the information already on the termination
notice, would ask for post-fellowship contact information, information on
thesis and published papers, and post-fellowship employment plans and
information.
4-2. Conduct Comprehensive External Studies: In addition to conducting
regular internal reviews of the program, as recommended in Chapter 2,
NCER could incorporate extensive external reviews of the program,
performed periodically. NSF subjects its fellowship program to a
comprehensive study by an external contractor every 10 years. Such reviews
expose the program to "outside" perspectives and recommendations that
may not be manifested through in-house studies. In this manner, the
18
-------
program can avoid being stove-piped into a certain mode of practice and will
be made more dynamic, as it incorporates up-to-date fellowship practices.
4-3. Track Research Grant Recipients: NCER could track the number of
fellows who go on to apply for and receive research grants from EPA in a
manner similar to that done by NIEHS. The process would be an in-house
tracking mechanism that could be easy to implement.
In relation to increasing diversity, we recommend that the Assistant
Administrator of ORD direct NCER to:
4-4. Increase Outreach Outside MAI. NCER could be more active in outreach
efforts to under-represented minorities (outside of its current MAI efforts) to
increase diversity in the applicant pool. There are several strategies used by
NSF to increase diversity that NCER can use, including focusing outreach
efforts toward minority-based professional organizations and toward under-
represented minority groups at top research schools across the country.
Agency Comments and OIG Response
ORD agreed to recommendations related to establishing performance measures
and the need for data collection and analyses, tracking the fellows after
completion of the program, and considering some of the procedures used by
other agencies. ORD prepared an action plan, with action officials and due dates
for each recommendation. However, ORD did not agree with our comments on
diversity and need for outreach to minorities outside of MAIs, as discussed in
Chapter 3. We have made some modifications to the report in response to
ORD's comments. ORD's comments are in Appendix D, and our evaluation of
those comments are in Appendix E.
19
-------
20
-------
Appendix A
Input for Logic Model
Depicting STAR Fellowship Program
The logic model presented in Chapter 1 places the STAR Fellowship Program in the larger
context of ORD and EPA goals. It shows what the program does and the resources used, and
illustrates the links between design, implementation, and outcomes. It also contrasts design
outcomes and program manager priorities. Logic models for the STAR and MAI fellowship
program are similar, which is why the logic model in Chapter 1 is only for the STAR Fellowship
Program. When there are differences between the two programs, they are both addressed in this
appendix.
Resources
Resources include the human and financial resources, and other inputs, needed to support the
program.
STAR Fellowships. EPA has funded approximately 100 STAR fellows a year, at
approximately $10 million per year, since 1995. The staff involved with the fellowship
program varies from year to year, with approximately seven project officers and two
managers. Also, support is provided by the EPA Grants Administration Division, which
administers the awards based on NCER's recommendations. Other resources include
contractors; universities (facilities and faculty); panels of scientists for evaluating the
applicants; and other Federal agencies that collaborate with NCER, such as NSF.
MAI Fellowships. The MAI Program provides up to $34,000 per year for a maximum of
2 years for Master's level fellows and 3 years for Ph.D. candidates. Since 1998, the
program has accepted 58 graduate fellows - 28 for Masters and 30 for Ph.D. programs.
The students attended 24 MAIs, which were historically Black, Native American,
Hispanic, and Alaskan Indian. The MAI Undergraduate Fellowship Program provides
funds up to $17,000 per year for fellows' junior and senior years; since 1995, the program
has accepted 90 fellows.
21
-------
Activities
Activities indicate the specific actions and tasks needed to produce the program's outputs. The
main activities of the fellowship program deal with outreach, selection, fellowship conferences,
close-out, and tracking.
Outreach. NCER announces the STAR Fellowship Program request for applications on
its web site and through posters and letters to universities, and at conferences.
Selection Process. NCER has established a two-step selection process consisting of an
external peer review and a relevance review. Once the applications are received there is a
process of peer review of applications by external experts. The reviewers consider
academic records, recommendations, and career goals of applicants. More applications
are marked "excellent" than EPA can fund. The final decision is made by NCER staff
according to such criteria as achieving a balance of fellowship among universities,
disciplines, and emphasizing applications in disciplines that EPA considers particularly
important to its science mission.
Funding and Renewal Process. NCER project officers prepare a funding package for
each candidate, and send it to the Grants Administration Division, with recommendations
to make appropriate fellowship awards. The duration of the STAR fellowship is not to
exceed 2 and 3 years for Master's degree and Ph.D. programs, respectively. According to
the terms of the agreement, students must furnish progress reports from their academic
advisors at the end of each year for renewal of the fellowship the following year. Project
officers collect annual progress reports from fellows' institutions and recommend
renewals to the Grants Administration Division.
Close-out and Tracking. The fellowship program was designed to track the fellows for
3 years after completing the fellowship. NCER is to track the fellows at the termination
of the fellowship by collecting completed Termination Forms from fellows. According to
the terms of the agreement, upon completion of the fellowship, the fellows must complete
and send to the Financial Management Center the Fellowship Termination Notice
(EPA Form 5770-9). A copy of that notice is to also be sent to NCER, along with any
thesis, dissertations, and published works that result from the fellowship program. The
form requires information on the type of fellowship, degree sought, date degree received
or expected, and mailing address after termination of fellowship. In addition, the form
also provides space for additional remarks, in which fellows or their sponsors can provide
feedback on how NCER can improve the operation of the EPA Fellowship Program.
Fellowship Conferences. Planning and arranging for annual conferences for the fellows
with guest speakers, and providing opportunities for fellows to share their research.
22
-------
Outputs
Outputs are the products, including goods and services, provided to the program's customers.
The outputs of NCER's activities are:
Awards For Environmental Studies and Research. NCER provides fellowship awards
for approximately 10 percent of all applicants through its highly competitive selection
process. NCER announces the request for application, and opportunities for fellowships
to the students, universities, and research institutions, which results in a large number of
applications from high quality students. Through MAI fellowships, NCER provides
funds to strengthen the capacity of minority institutions to participate in environmental
studies and research.
Annual Conferences for Fellows. NCER has held six annual conferences for fellows
since 1996. The proceedings of the annual meetings are published as a detailed report.
The conference provides opportunities for fellows to meet with other fellows, share their
project information through posters, and meet with representatives from EPA program
offices. Program office representatives can look for fellows suitable for recruitment into
EPA upon completion of the fellowship.
Progress Data on Fellows. NCER collects progress reports to gather information on
fellows meeting academic requirements, before continuing next year's funding. They
track the fellows through completion of the program, by collecting the termination forms
and papers and thesis completed.
Post-Completion Data. The fellowship program is designed to track the fellows after
completion, in order to ascertain whether they entered an environmental career after
fellowship. The fellowship terms and conditions require the fellows to agree to notify the
EPA project officer of all employment for at least 3 years after graduation. The program
can consolidate the information on the termination forms into a database of degree
completion, thesis completed, jobs, careers chosen, and post-fellowship contact
information. Such a database can be of great value to EPA program offices, researchers,
and industry, as potential researchers and employers.
Customers
Customers are the users of the program's outputs. The customers of the STAR Fellowship
Program include students and universities, to further education opportunities; and EPA program
offices, other Federal offices, State and local governments, industry, and non-governmental
organizations, which are potential employers of the future environmental specialists coming out
of the fellowship program.
23
-------
Outcomes
Short-term Outcomes. These outcomes indicate changes in customers knowledge,
skills, and aspirations, followed by changes in customer actions or behavior. The short-
term outcome of the fellowship program is an increased number of people with graduate-
level degrees in environmental fields, who continue in higher studies or research, or work
in government, industry or academia.
Intermediate Outcomes. These outcomes can be defined as environmental changes
resulting from customer actions. Intermediate outcomes of the fellowship program would
result from more informed policy decisions and solutions to current and future
environmental issues, provided by a steady stream of well-trained environmental
specialists nationwide. As part of being a national leader in environmental research,
ORD has a role in developing the next generation of environmental specialists.
Long-Term Outcomes. Such an outcome is to support the overall EPA mission of
providing better human health and a healthier ecosystem. Continued use of sound
science, better policy in government and the private sector, over a long period of time,
will result in a cleaner environment and improved human and ecosystem health.
Externalities
Externalities are factors that impact the program but are outside its control. The program needs
to be aware of them. Although the program cannot control these factors, they can influence some
of them. Some of the externalities for the fellowship program are Congress; Office of
Management and Budget (OMB); other Federal agencies' programs; universities and other
academic institutions; and other factors, including the national economy.
Performance Measures
A program design must include certain meaningful measures that can show the progress toward
or attainment of results. Need for performance measures has been emphasized by the
Government Performance and Results Act and OMB, to ensure accountability of Federal
programs. As noted in Chapter 2, NCER has not established clearly defined performance
measures for the fellowship program, which is why no entries for performance measures are
included in the logic model provided. NCER is in the process of developing performance
measures.
24
-------
Appendix B
Details on Scope and Methodology
To determine the degree to which the objectives of the STAR and MAI Fellowship Programs
were met, and how they contributed toward ORD's and EPA's goals, we did the following in
three areas.
(1) To identify the specific performance measures used by EPA/ORD/NCER to evaluate the
STAR Fellowship Program, we:
Reviewed the design of the fellowship program for its logic and completeness.
We used the logic model as a tool to document the full spectrum of the fellowship
program, documenting the resources; program activities and processes; outputs;
customers using those outputs; and short-term, intermediate, and long-term outcomes.
We obtained information from the web site, through interviews with NCER staff and
managers, and by reviewing program files. We shared the logic model (Figure 1-1,
Chapter 1) and explanatory notes (Appendix A) with the NCER managers, and have
incorporated their comments as appropriate.
To understand the processes used by NCER to implement the program, obtained
information from discussions with NCER managers and staff, who also provided us
with data and files on fellows. We met with four NCER project officers, the Director
of the STAR Fellowship Program, and the Director of NCER.
Reviewed a randomly selected representative sample of 50 fellowship files - 10 files
for each year - out of a total of 587 files from 1995 through 1999. We selected seven
markers of progress and indicators of success that cover the total program from
beginning (Resources and Activities) to the end (Results/Outcomes) and used these
markers for evaluation. These markers were: Progress Report #1 at the end of year
one, Progress Report #2 at the end of year two in a 3-year fellowship, completed
Termination Notice (EPA Form 5770-9), Degree Completed, Thesis/Papers
Published, Post-Fellowship Information, and Career Information (after 3 years).
Coordinated with the NRC staff conducting the review of STAR Research Grants,
and attended the public sessions at the NRC, to share the information and discussions
about the STAR Program. The discussions included presentations made by EPA
managers, as well as presentations made by representatives from NSF, NIEHS,
Department of Energy, and Department of Agriculture, who have environmental and
other research grants and fellowship programs similar to EPA's STAR Research
Grants Program
Reviewed the NRC report issued at the completion of the review of EPA's STAR
Grants Program and noted the main approach and recommendations provided.
25
-------
To identify the diversity of the applicant pool, and compare it with the demographics of
the fellowship recipients, we:
Requested demographic data from the NCER staff responsible for maintaining the
demographic profile of the STAR Fellowship Program applicant pool and those who
received a fellowship award. The information is provided by applicants voluntarily,
resulting in many gaps in the data. We analyzed the data received. We were not able
to perform trend analyses on participation of under-represented minority groups
because NCER did not have data for years prior to 1998, and there were many gaps in
the data since 1999 (Chapter 3). We note a scope limitation due to lack of data
availability.
Requested similar demographic information regarding the MAI fellowship, regarding
race and gender, because NCER uses the MAI fellowship to address diversity. We
reported the data received.
Have presented the demographic data for both the STAR and MAI programs
(Chapter 3), as provided by NCER. We did not independently test or verify the data
provided by NCER.
Reviewed literature on workforce analyses and on under-representation of women and
minorities in S&E labor force, and on the business need for increasing diversity in the
S&E labor force.
To identify and compare the STAR Fellowship Program with similar fellowship programs
at other Federal agencies, we:
Reviewed fellowship programs of agencies with established fellowship programs, and
followed up with the project officers from DOD, NSF, and NIEHS, to obtain and
document the procedures developed and used by these agencies and provide
recommendations to EPA as appropriate. The DOD program reviewed is managed by
the American Society for Engineering Education, a non-profit organization. NIEHS
is an institute within the National Institutes of Health.
Researched the literature of available evaluations of fellowship programs - some
public and some private - and literature on the feasibility of evaluating research
programs, as well as the appropriate criteria for such programs, as recommended by
OMB, National Academies, and academic evaluators.
26
-------
Appendix C
Additional Information on
Other Fellowship Programs
Selection Process
All of the fellowship programs reviewed have incorporated peer review panel
systems for their selection processes, and each has unique criterions in place to
select only the top candidates (DOD: academic scores; NSF: two-merit criteria).
EPA's STAR Fellowship Program uses peer reviewers in the selection process.
EPA also uses many criterions in selecting awardees, including: geographic
distribution across States, distribution among universities, projected
environmental workforce needs, relevance of research proposal to EPA's
mission, and availability of funds. A summary of the selection process for the
three other programs is in Table C-l.
Table C-1: Selection Processes for Other Agency Programs
DOD
NSF
NIEHS
•	120 professionals review and
rank applications
•	Highly-ranked list sent to DOD
for selection
•	Selection based on academic
scores
•	Utilizes two merit criteria
("intellectual" and "broader
impacts" criterions)
•	Contractors conduct initial
screening to check for
completeness and eligibility
•	Each application is read and
graded by two independent
experts from a panel and the
top 40 percent are read over
by a third expert
Selects academic institutions/
programs to dispense grants to
individual students (selection
based on peer reviews, priority
scores, institution's plans for
recruitment and retention of
minorities, and the quality of the
institution)
Project Officer Duties
All the programs have project officers working directly with the fellowship
programs to track progress of fellows and administer funding during the duration
of the individual fellowships. Compared to the fellowships reviewed, EPA's
STAR Fellowship Program has the most project officers involved in the
fellowship program (six to eight). The STAR Fellowship Program pairs fellows
27
-------
to appropriate project officers in accordance with the officers' background and
expertise and the topic area of fellows' research. Details are in Table C-2.
Table C-2: Project Officer Duties for Other Agency Programs
DOD
NSF
NIEHS
•	One project manager works
with two to three assistants
•	Tracks fellows' academic
progress during the fellowship
by receiving transcripts from
schools and progress reports
from academic advisors
•	Two project officers, one
deals with application review
process and the other with
post-award administration
•	Heavily involved with
administering funds with
institutions and fellows and
collecting Activities Reports
•	Number of project officers not
available
•	Select academic institutions
and research programs based
on peer reviews, priority
scores, etc., and provide for
extensions when needed
28
-------
Appendix D
^DSr-%
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
RESEARCH AND DEVELOPMENT
September 15, 2003
MEMORANDUM
SUBJECT: Response to the Office of Inspector General's Draft Audit Report #2001.001545
FROM: Paul Gilman /s/ Paul Gilman
Assistant Administrator (8101R)
Thank you for the opportunity to review the draft evaluation report #2001.001545. I do
not concur with the analysis or conclusions on diversity in the report. The report should be
substantially rewritten to remove any discussion on diversity and Chapter Three should be
withdrawn. (See OIG Response 1, Appendix E) I concur with the findings and
recommendations that pertain to performance measures in your assessment of the Science to
Achieve Results (STAR) Fellowship Programs that are managed by the National Center for
Environmental Research (NCER.) These are improvements we began implementing before the
program was eliminated from the agency's budget for 2002.
We do not concur with the findings, recommendations, or results-in-brief in the report
related to diversity. A primary basis for our nonconcurrence is the incorrect assertion in the
report that one measure of effectiveness of the STAR Fellowship Program should be "increasing
the diversity in the next generation of the environmental work force" (see the Purpose section at
page 1, lines 5-6.) The report correctly acknowledges that increasing diversity in the applicant
pool is not a specific goal of the STAR Fellowship Program, but then illogically concludes that
the program has not achieved this outcome. Measuring the performance of the fellowship
TO:
Jeffrey K. Harris
Director for Program Evaluation
Cross-Media Issues
Office of Inspector General (2460T)
Diversity
29
-------
program in an area that it does not have as a goal is analogous to failing a student for a course he
never took.( See OIG Response 2, Appendix E)
The obstacles to increased diversity at the graduate level begin very early in the education
process. Solving these problems will require efforts at all educational levels in this country and
cannot be solely remedied by programs which deal with the end of the educational pipeline. The
applications to the STAR program must come from the existing population of science and
engineering graduate students. Though we are limited by the current ethnic/racial distribution of
students in science and engineering programs (S&E), we actually attract a more diverse group of
fellows than the population in the S&E labor force. Data cited in the draft report illustrate this
point: "Women comprise 46 percent of the workforce but only 23 percent of the S&E labor force.
African Americans and other ethnic minorities constitute 24 percent of the total population but
only 7 percent of the S&E labor force." The population of our fellows is more diverse, with an
average of approximately 60% women and 12% minority students in the STAR program in the
most recent year, and about 70% women and 58% minority students in the combined Minority
Academic Institutions (MAI) programs.(See OIG Response 3, Appendix E)
We have always been committed to making this program available and accessible to
minority students. (See OIG Response 4, Appendix E) For example, we currently conduct
considerable outreach to ensure that under-represented groups are aware of all of our programs,
including the STAR and MAI fellowships. We have recruited additional minority reviewers as
part of our peer review process. We have targeted outreach efforts for both programs to the
institutions and professional organizations that serve minorities because the percentage of under-
represented groups at these organizations is much higher than in the total pool of science
graduate students.
In 1999, we revised the MAI graduate fellowship program to mirror the process and
provisions of the STAR program. We also increased the number of fellowships awarded. These
changes were intended to increase capacity at MAIs as well as awareness of under-represented
groups. The chapter as written does not recognize the MAI programs as valid outreach efforts to
under-represented groups. We believe that the MAI programs are valid and effective outreach
programs, and they represent the most efficient use of staff time for outreach to minority groups.
(See OIG Response 5, Appendix E)
We agree with the recommendation to collect and analyze demographic data on our
applicants and fellows and we will evaluate other federal efforts in this regard. (See OIG
Response 6, Appendix E) We recommend eliminating the diversity chapter of the report and
including the collection and analysis of demographic information among the activities you
recommend in Chapter Two under Performance Measures.(See OIG Response 7, Appendix E)
Performance Measures
OIG found that NCER has focused its efforts on developing excellent solicitations and
selecting the best candidates to receive fellowships. The report recommends that NCER should
place additional emphasis on reporting outcomes of the program. Now that the program is
30
-------
established and running smoothly, we agree that we should tie our performance measures more to
outcomes. We began such efforts in 2001, but halted them when the program was eliminated
from the Agency's 2002 budget. With the program operational once more, we are again focused
on this task.
Chapter Two makes three recommendations pertaining to performance measures:
1.	Conduct Internal Evaluations: ORD completed the first such evaluation in May 2003 .(See
OIG Response 8-A, Appendix E) One of the first outcomes of that internal evaluation was a
series of contacts with program offices and regions, to gain their perspective on the disciplines
that will be needed to address emerging science questions in the future. We have also redoubled
our efforts to develop an effective post-fellowship tracking system that can be managed with the
resources currently available. Another result of the internal evaluation was an analysis of the
fellowship provisions and recommendations for changes that make the program more
competitive with other federal programs.
2.	Establish Performance Measures: ORD agrees with the recommendation to establish
performance measures and to ensure that we track them. We also agree with the measures that
you suggest we track, including successful completion of the degree program, relevant papers
published, and postgraduate pursuit of an environmentally related career. We had included these
measures in our previous tracking plans. (See OIG Response 8-B, Appendix E)
3.	Collect Data: The report discusses data collection at two points in the program: (1) during
the fellowship, including termination, and (2) after the fellowship, including three years of
follow-up on graduation and career progress.
A.	Data Collection During the Fellowship: OIG staff studied 50 files for STAR
recipients during the years 1995 - 1999, the first four years of the program, and reported
that the files usually contained annual reports, but often lacked information regarding
graduation dates, publications, and post-fellowship activities. The following information
is currently collected from all students who are concluding their fellowships:
a.	Name of degree and date expected or conferred.
b.	Name and address of post-fellowship employer or postdoctoral advisor.
c.	Post-fellowship contact information, including address, telephone, e-mail, and
alternate contact person.
d.	Electronic copies of all publications, including dissertations, published,
submitted, or in press.
e.	A full bibliography of publications over the life of the fellowship.
This information is compiled into a master spreadsheet. At present, fellows are followed
for at least one year after they leave the program and NCER is investigating ways to
follow them over the long term.(See OIG Response 8-C, Appendix E)
B.	Data Collection After the Fellowship: We agree that fellows should be followed
closely after they complete the fellowship, to determine whether they continue in
31
-------
environmental careers after graduation. We are investigating procedures used by other
federal agencies in order to collect this information for the STAR fellowship recipients
and comply with existing regulations, such as the Paperwork Reduction Act.(See OIG
Response 8-D, Appendix E)
How Other Agencies Evaluate Effectiveness:
We appreciate the work of your staff to locate similar fellowship funding organizations
and compare approaches. Several of the suggestions made in the draft report appear workable for
the STAR Fellowship Program. In particular, we like the idea of tracking fellows who go on to
obtain STAR grant awards. We also like NSF's approach of conducting an extensive evaluation
of the fellowship program every ten years. Since the STAR program will reach the ten-year mark
in 2005, we have begun planning for the first such review of our program. As part of that
process, we will work with representatives from the other agencies to develop and implement
procedures that would enhance the operation of the STAR program. (See OIG Response 9,
Appendix E)
Attachment 1 contains our consolidated, page-specific comments, which we offer for
incorporation in the final report. We have included our corrective action plan as Attachment 2 to
the draft's recommendations. The final report should reflect those recommended actions that
ORD has already completed. If the final report removes all discussion on diversity and the
corresponding recommendation as we have requested, we believe the report can be closed out
upon issuance.
Thank you for the opportunity to review and respond to this draft report. Should your
staff have questions or require further information, please have them contact Arnold Bloom at
202-564-6687.
Attachments (2)
32
-------
Appendix E
OIG Response to Agency Comments on Draft Report
We have provided responses to the main comments provided by the Agency. The text of Agency
response is provided as Appendix D. The Agency comments are numbered and in bold italics,
followed by the OIG responses.
1.	The report should be substantially rewritten to remove any discussion on diversity and
Chapter Three should be withdrawn.
We have revised the draft report to incorporate some of the Agency's suggestions. We
consider the discussion of the demographic composition of the applicants and fellows a
relevant consideration within EPA's commitment to having a diverse workforce. The issues
related to demographics and diversity are discussed in more detail below in Nos. 2 through 7.
Diversity ( Chapter 3)
2.	A primary basis for our nonconcurrence is the incorrect assertion in the report that one
measure of effectiveness of the STAR Fellowship Program should be "increasing the diversity
in the next generation of the environmental work force" (see the Purpose section at page 1,
lines 5-6.) The report correctly acknowledges that increasing diversity in the applicant pool is
not a specific goal of the STAR Fellowship Program, but then illogically concludes that the
program has not achieved this outcome. Measuring the performance of the fellowship
program in an area that it does not have as a goal is analogous to failing a student for a
course he never took.
We modified the language of the report to clarify our objectives for looking for demographic
information on the STAR applicants and fellows. Our objective was to provide a
demographic profile of the applicants and fellows, since EPA has a commitment to have a
diverse workforce, and the goal of the STAR Fellowship Program is to develop the
environmental work force of the future. However, despite repeated efforts, we were not able
to obtain sufficient demographic information from program managers. No data were
collected prior to 1999, and since 1999 data have not been collected consistently. The lack of
data from 1995-1998 and gaps in the data even in recent years indicates that NCER did not
place an emphasis on the collection and documentation of participant demographic
characteristics.
Increasing diversity in fellowship programs is considered important by all three of the other
fellowship programs we reviewed (Chapter 4). Two of the three programs make increasing
diversity a stated objective or criteria. All three of them collect and analyze the data, and
adjust outreach efforts based on the results of analyses. None of them limit outreach to MAI;
each seeks minority applicants from top-ranking universities.
33
-------
The draft report acknowledged that increasing diversity was not a stated objective of the
STAR Fellowship Program, and recommended collecting demographic data so that NCER
can evaluate the demographic composition of the applicants and fellows. The conclusion was
not that NCER failed in increasing diversity, but rather that the lack of adequate demographic
data inhibits the program from determining whether it has been successful in attracting and
selecting a diverse pool of applicants and making the program accessible and available to
minority students.
3.	The population of our fellows is more diverse, with an average of approximately 60%
women and 12% minority students in the STAR program in the most recent year, and about
70% women and 58% minority students in the combined Minority Academic Institutions
(MAI) programs.
The data on which these statements are based was never shared with the OIG. In order to
complete our analyses, we requested demographic data on applicants and recipients on
several occasions from NCER Managers. We have presented the data we received in tables
and figures in Chapter 3. Figure 3-3 shows combined numbers of applicants as about
54 percent female, 42 percent male, and 4 percent unknown. Figure 3- 4 shows the data on
recipients as about 28 percent female, 21 percent male, and 51 percent unknown.
4.	We have always been committed to making this program available and accessible to
minority students.
NCER can demonstrate this commitment clearly if it collects data and maintains a database
on applicants and fellows. The issue we highlight in the report is the need for collecting,
maintaining, and analyzing demographic data.
5.	The chapter as written does not recognize the MAI programs as valid outreach efforts to
under-represented groups. We believe that the MAI programs are valid and effective outreach
programs, and they represent the most efficient use of staff time for outreach to minority
groups.
The fellowships awarded to MAI in response to the White House Initiative serve a useful
purpose of improving the quality of education at institutions that have historically provided
education to minority groups. There is a higher number of minorities in schools recognized
as MAI. This program has been in existence since 1981, before the STAR program was
established. In addition to the contribution of the MAI, NCER might be able to attract
additional high caliber minority applicants from top ranking institutions. The brightest
minority candidates, who may not be enrolled in an MAI, are ineligible for the MAI
Fellowships. Eligibility for a MAI Fellowship depends on the institution being recognized as
MAI, rather than the student being from a minority background. As illustrated in Table 3-3,
in 2002, only 5 of the 14 MAI fellows (where race/ethnicity data was recorded) were
non-white.
34
-------
6.	We agree with the recommendation to collect and analyze demographic data on our
applicants andfellows and we will evaluate other federal efforts in this regard.
We note that ORD agreed with our main point - that it needs better demographic data.
7.	We recommend eliminating the diversity chapter of the report and including the collection
and analysis of demographic information among the activities you recommend in Chapter
Two under Performance Measures.
We concluded that a discussion of the need to collect and analyze participant demographic
data in Chapter 2 under Performance Measures could lead to the perception that increasing
diversity was an explicit performance goal on the STAR Fellowship Program. We chose not
to alter the draft report format but to modify the text in Chapter 3 to address concerns.
Performance Measures (Chapter 2)
8-A. Chapter Two makes three recommendations pertaining to performance measures:
1. Conduct Internal Evaluations: ORD completed the first such evaluation in May 2003.
This information was not shared with us prior to the response to draft report, even at our
June 2003 exit meeting with NCER managers.
8-B. 2. Establish Performance Measures: ORD agrees with the recommendation to
establish performance measures and to ensure that we track them We also agree with the
measures that you suggest we track, including successful completion of the degree program,
relevant papers published, and postgraduate pursuit of an environmentally related career. We
had included these measures in our previous tracking plans.
The Agency agrees with the recommendation. The Corrective Action plan has designated an
Action Official and a due date of January 2004. The previous tracking plans were not
discussed at any of the meetings during our field work.
8-C. Data Collection During the Fellowship.
... The following information is currently collectedfrom all students who are
concluding their fellowships:
a.	Name of degree and date expected or conferred.
b.	Name and address ofpost-fellowship employer or postdoctoral advisor.
c.	Post-fellowship contact information, including address, telephone, e-mail, and
alternate contact person.
d.	Electronic copies of all publications, including dissertations, published, submitted,
or in press.
e.	A full bibliography ofpublications over the life of the fellowship.
35
-------
This information is compiled into a master spreadsheet. At present, fellows are followed for at
least one year after they leave the program and NCER is investigating ways to follow them
over the long term
During our field work, NCER provided five spreadsheets - one for 1995-1998 and one each
for 1999, 2000, 2001, and 2002 (MAI only). The spreadsheet for 1995-1998 had very little
useful information on items a, b, c, d, or e. The spreadsheets from 1999, 2000, 2001, and
2002 had information on item a. (name of degree and date expected or conferred), but there
was no information on b, c, d, or e. NCER did not share with us any spreadsheets that
collected name and address of post fellowship employers or post doctoral advisors. The
Termination Letter also asks for some followup information, but we did not see any
spreadsheet completed from the information based on that information.
8-D. Data Collection After the Fellowship: We agree that fellows should be followed closely
after they complete the fellowship, to determine whether they continue in environmental
careers after graduation. We are investigating procedures used by other federal agencies in
order to collect this information for the STAR fellowship recipients and comply with existing
regulations, such as the Paperwork Reduction Act.
The Agency agrees with the recommendation to collect data during, and track the careers of,
the fellows after completion of fellowship. The goal of the program is to encourage students
to obtain advanced degrees, and pursue careers in environmentally related fields. In order to
measure the outcomes of the program, it is essential to track the fellows through degree
completions, and track the careers followed upon completion.
How Other Agencies Evaluate Effectiveness (Chapter 4)
9. We appreciate the work of your staff to locate similar fellowship funding organizations and
compare approaches. Several of the suggestions made in the draft report appear workable for
the STAR Fellowship Program
The Agency agrees with some of the approaches used by other agencies for tracking the
fellows after completing the fellowship program. The information gathered by NCER at the
time of completion in the tracking form should also be organized in a database that can be
used for analyses and program evaluation. At the time of field work, the information was
stored in individual folders of the fellows, but there was no database that could provide the
information provided in the Termination Forms.
36
-------
Appendix F
Distribution
Assistant Administrator, Office of Research and Development (8101R)
Comptroller (2731 A)
Agency Followup Official (271 OA)
Agency Followup Coordinator (2724A)
Audit Followup Coordinator, Office of Research and Development (8102)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Communications, Education, and Media Relations (1101 A)
Acting Director, National Center for Environmental Research, ORD (8701)
Acting Deputy Director, National Center for Environmental Research, ORD (8101R)
Director, Environmental Sciences Research Division, National Center for Environmental
Research, ORD (8723R)
Inspector General (2410)
37
-------