v>EPA
United States
Environmental Protection
Agency
Office of
Solid Waste and
Emergency Response
DIRECTIVE NUMBER: 9028. OOa
TITLE. OSIR1 S SIS1EH LIFE CICLE MANA01ENT GUIDANCE; PAR! THREE m SIX PRACTICE PAPERS
Project Participation and Coordination; Systei Life Cycle Reviews and Approvals;
Systei Life Cycle for Expert .Systems; Data Hanapent; Configuration Management;
APPROVAL DATE:
EFFECTIVE DATE:
ORIGINATING OFFICE:
0 FINAL
D DRAFT
STATUS:
REFERENCE (other documents):
OSWER OSWER OSWER
fE DIRECTIVE DIRECTIVE
-------
unites states environmental protection Mgency
Washington. DC 20460
OSWER Directive initiation Request
1. Directive Number
9028.OOa
2. Originator Information
vjame of Contact Person
Asa R. Frost, Jr.
Mail Code
OS-110
Office
OPMT/IMS
Telephone Code
47506760
3. Title
OSWER'S SYSTEM LIFE CYCLE MANAGEMENT GUIDANCE: PART THREE _ SIX PRACTICE PAPERS
Project Participation and Coordination; System Life Cycle Reviews and Approvals;
System Life Cycle for Expert .Systems; Data Management; Configuration Management;
-t.tSoajieiytDf BimBagdmdadB tffiitetaiemeru of purpose)
The Guidance for information systems provides a structured approach for the solution of
information management problems, particularly those that require consideration of
automated systems. The Guidance explains the importance of life cycle management and
describes the progression of the life cycle in terms of the activities and products '
required for each phase of the life cycle. .'''"
5. Keywords
Automation, baseline information standards, data, data management, guidance
6a. Does This Directive Supersede Previous Oirective(S)? I J( I
b. Does It Supplement Previous Directive(s)?
Yes What directive (number, title)
Yes What directive (number, title) 9028 00
OSWER1s System Life Cycle Mgmt.Gc
7. Draft Level
A - Signed by AA/DAA
B - Signed by Office Director
! - For Review & Comment
D - In Development
8. Document to be distributed to States by Headquarters?
Yes
X
No
This Request Meets OSWER Directives System Format Standards.
9. Signature of Lead Office Directives Coordinator
Date
10.
Date
EPA ForrrMef5-17 (Rev. 5-87) Previous editions are obsolete.
OSWER OSWER OSWER O
VE DIRECTIVE DIRECTIVE DIRECTIVE
-------
I" UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
= fc""r- WASHINGTON, D.C. 20460
MAR 2 2 IQOO
OFFICE OF
SOLID WASTE AND EMERGENCY RESPONSE
MEMORANDUM
SUBJECT: Part Three Of OSWER's System Life Cycle
Management Guidance - Directive No. 90^
FROM: Asa R. Frost Jr., Director
Information Management Staf
TO : Addressees
fj^n
f F"~
OSWER Directive 9028.00 states that every information
management project which supports the Agency's hazardous waste
program must comply with OSWER's System Life Cycle Management
Guidance. Parts 1 and 2 of the Guidance were distributed several
months ago. Part 3 is reserved for Practice Papers which describe
a particular topic in depth, reflecting changes in technology or
organizational structures. Practice Papers on six topics have been
completed and are now being issued as an addition to Directive
9028.00. The set of Practice Papers is available in Room M2416 of
Headquarters, or by mail by calling FTS 475-6760. In addition,
the set of Practice Papers is being distributed to those who have
already received Parts 1 and 2 of the Guidance.
The Guidance serves to promote better management of
information activities by presenting a structured approach for
solving problems and for selecting and using the methods, tools
and technology appropriate to each project. The Practice Papers
in Part 3 cover the following topics:
"Data Management During the Life Cycle" provides details to
project managers concerning data modeling, data design, data
documentation, and such key concepts as data stewardship and
custodianship. It is the basis for OSWER's data administration
program.
"Project Management Plan" describes the structure and content
of a key document of the system life cycle.
"Configuration Management" describes OSWER's practice of
change control and accounting during the system life cycle.
-------
"Systra Life Cycle Reviews and Approvals" describes the
process for obtaining reviews and approvals of life cycle
products from the appropriate levels of organization such as
the OSWER Information Management Steering Committee.
"System Life Cycle Paper for Expert Systems" provides guidance
on the design, development and operational issues for expert
systems.
"Project Participation and Coordination" describes the typical
roles to be filled during the system life cycle and suggests
which parts of the organizations should fill those roles.
If you have any questions concerning the Guidance, or the
Practice Papers, please call me or Mary Lou Melley at FTS 475-6760.
-------
Addressees (without Attachment)
OSWER Information Management Steering Committee Members
OSWER Division Directors
OSWER Program Management Staff Directors
OSWER Staff Directors, Office of Program Management and Technology
Headquarters SIRMOS
Inspector General
Assistant Regional Administrators
Regional Planning Coordinators for Hazardous Waste
Regional SIRMOS
ORD Headquarters and Laboratory Managers
OSWER Contractors
OIRM Contractors
Addressees (with Attachment)
OSWER Information Management Coordinators
OIRM Division Directors
National Data Processing Division Director
Regional Waste Management Division Directors
Regional IRM Branch Chiefs
Individual recipients of OSWER's Life Cycle Management Guidance,
(Parts 1 and 2)
-------
REVIEWS AND APPROVALS
-------
OSWER DIRECTIVE #902B.OOa
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
* .
Part 3: Practice Paper
System Life Cycle
Reviews and Approvals
January, 1989
-------
OSWER DIRECTIVE *9323.COe
TABLE OF CONTENTS
Page
1. Practice Paper Purpose 1
2. Application of Life Cycle Management Guidance 1
3. Review and Approval Process 2
3.1. Overview 2
3.2. Reviews 4
3.3. Approvals 6
4. Threshold Analysis -. 8
4.1. Purpose of Threshold Analysis 8
4.2. Performing Threshold Analysis 8
EXHIBITS
3-1. Review and Approval Process for Each Stage :... 3
4-1. Criteria for Reviews and Approvals 9
4-2. Threshold Analysis Process 10
-------
OSWER DIRECTIVE «5QSB.OOa
1. PRACTICE PAPER PURPOSE
This practice paper constitutes a section of Part 3 of the
Office of Solid Waste and Emergency Response (OSWER) System Life
Cycle- Management Guidance. It describes the review and approval
process noted in Guidance Parts 1 and 2 in the context of the
OSWER program and information management organizations.
The topics addressed in this practice paper include:
o The scope of application of OSWER's Life Cycle Management
Guidance;
o The OSWER review and approval process in a system life
cycle; and
o OSWER1s threshold analysis methodology for determining the
appropriate review and approval levels for each project.
OSWER's process for review and approval of system life cycle
efforts has seven primary purposes:
o To confirm the information management requirements for the
system:
o To confirm the quality of the life cycle products from a
programmatic, technical, and project management
perspective;
o To confirm that all programmatic, technical, and project
management issues have been appropriately resolved, and at
the proper time;
o To confirm and document*program management commitment to
the project and its management approach (defined
requirements, schedules, plans, resources, and budget);
o To approve continuation with the next stage of the life
cycle;
o To confirm the continued commitment of resources to the
project for the remainder of the life cycle; and
o To provide additional feedback to the project team.
2. APPLICATION OF LIFE CYCLE MANAGEMENT GUIDANCE
This Guidance applies to systems intended for use by an OSWER
organization, op by a Regional office or state agency in support
of an OSWER program. It includes systems funded by the.direct
-------
OSWER DIRECTIVE #3023.033
OSWER budget, or by other EPA offices in support of OSWER. It
excludes systems developed by state agencies for their own use,
and systems developed by OIRM and ORD for Agency-wide use, even
if they support or otherwise impact OSWER programs or resources.
It includes systems developed under OSWER mission support
contracts, even those incidental to the major purpose of a
contract.
Although the Guidance has broad application, it provides the
flexibility needed to address a wide range of information
systems. It does not prescribe a single method, or present a
"cookbook" approach applicable without change to every
information management need or system. Rather, it presents part
of a structured, disciplined approach for solving information
management problems, and for selecting and using the methods,
tools, and techniques appropriate to each problem. The basic
life cycle management approach is tailored to each system, and is
documented in the Project Management Plan. (See OSWER Life Cycle
Management Guidance, Part 3, Practice Paper; Project Management
Plan.)
3. REVIEW AND APPROVAL PROCESS
3.1. Overview
Project activities and products must be reviewed and
approved at each stage of the system life cycle. Exhibit 3-1
illustrates the structure and timing of the review -and approval
process. Four important aspects of the review and approval
process are noted below:
o Threshold Analysis determines the appropriate review and
approval levels for a system, clearly identifying those
systems that require approval from the OSWER Information
Management Steering Committee. It is first performed in
the Initiation phase, and is updated in each subsequent
phase or stage as needed.
o In most stages, a 'project awaiting approval may continue
to the subsequent stage. If the project completes that
stage while the earlier approval is still pending, work
stops until the pending approval is received unless the
project manager obtains a written waiver.
o Before a project is submitted for approval by the OSWER
Information Management Steering Committee, the OSWER
Senior Information Resources Management Officer (SIRMO)
and Information Management Coordinator(s) (IMC(s))
involved in the project conduct the required reviews and
document their findings. Based on the results of the
review, the Project Manager' determines the specific
-------
EXHIBIT 3-1:
REVIEW AND APPROVAL PROCESS FOR EACH STAGE
BEGIN
STAGE
.
} PARALLEL I
REVIEW I
| (OPTIONAL) I
\
PROJECT
APPROACH AND
EXECUTION
ACTIVITIES
THRESHOLD
ANALYSIS
FEEDBACK
""..
PREPARE
SYSTEM
DECISION
PAPER
*.
REVIEW AND APPROVAL THRESHOLDS
Level I
Level
Review
Approval
IMC(s) and OSWER S1RMO IMC
Designated Office Directors) Office Direclor
and 1M Steering Committee
a
I !
a
ri
-------
changes or adjustments to be made, and determines when
to proceed to the OSWER Information Management Steering
Committee to obtain the required approvals.
o No system may be fully implemented (i.e., progress from
the Implementation stage to the Production stage)
without formal approval.
3.2. Reviews
"^"^^^^^^^^^MV 4
3.2.1. Purpose of Reviews
Reviews are formal examinations of life cycle products to
verify that they solve the information management problem.
Reviews are advisory activities, not decision-making activities.
They are advisory to the approval authority and to the project
team. They have four major purposes:
o To confirm the continuing validity of the system
requirements;
o To confirm that the contents of the system's
programmatic, technical, and project management
products reflect the requirements;
o To confirm that all programmatic, technical, and
project management issues have been appropriately
resolved; and
o To provide additional feedback to the project team.
3.2.2. Identification of Needed Reviews
Reviews are conducted prior to all approvals. Each
review to be conducted is identified in the Project Management
Plan, with the timing, content, and participants noted for each
review. Where needed, the content and timing of each review is
tailored to reflect any consolidation of life cycle stages.
3.2.3. Reviewers
/
The principal (or lead) reviewer is determined by the
results of the analysis of the review criteria (see Section 4),
and is either an IMC or the SIRMO. When the SIRMO is the
reviewer, the IMC(s) of the program(s) also participate in the
reviews. Office Directors may establish additional review levels
within their own offices. The lead reviewer is identified in the
Project Management Plan.
The lead reviewer is supported by other organizations or
individuals (e.g., representatives of OSWER, Regional offices,
state agencies/associations, OIRM, contractors), as determined by
the reviewer. The supporting reviewers are identified in the
Project Management Plan.
-------
OSWER DIRECTIVE r;2Q22.C2a
3.2.4. Conduct of Reviews
Reviews may be conducted during and at the completion of
project execution activities (i.e., the activities performed by
the project team), or both. Reviews that are performed during
project execution activities may include attendance by the
reviewer(s) at key project meetings or presentations, as well as
an examination of individual life cycle products. The timing and
scheduling of reviews are determined jointly by the reviewer and
Project Manager, and are documented in the Project Management
Plan.
Reviews examine (directly or indirectly) all products of
the life cycle, and always include an examination of the system
decision papers. Reviews are carried out in two steps,
reflecting the reviewer's confidence in the implementation of
life cycle management and the strength of project management:
1. The reviewer examines the* project management products
to confirm that the life cycle management activities
of the current phase or stage have been planned and
conducted appropriately. In such a case, the
reviewer may elect to conduct a high-level review of
most other products, and examine only the system
decision paper in detail.
2. In other cases, the reviewer may examine in detail
the content of one or more of the other life cycle
products of the current phase of stage.
Review results are discussed with the project team, and
provide valuable feedback. Individual life cycle products are
revised and reviewed as appropriate before submission to the
approval authority. The Project Manager determines those
modifications that are appropriate. For a project that requires
the approval of the OSWER Information Management .Steering
Committee, the OSWER SIRMO and IMC(s) involved in the project
conduct the review and document their findings. Although the
Project Manager* and the reviewers should agree on the changes
needed, if any prior to submission of the system decision paper
for approval, such agreement is not mandatory. The Project
Manager will determine the changes to be made, and when to
request approval from the OSWER Information Management Steering
Committee.
3.2.5. Timing of Reviews
The length of time required for reviews varies; parallel
review should be employed for particularly large projects or
projects with very tight schedules. The duration for each review
should be determined jointly by the reviewer and project manager.
In any case, review(s) are to be completed before the products
are submitted to the approval authority. For systems requiring
OSWER Information Management Steering Committee approval, the
-------
review should be completed, and the pertinent products and review
findings provided to the OSWER Information Management Steering
Committee, at least two weeks prior to the scheduled committee
meeting date (or the date for written concurrence if no meeting
is scheduled).
3.2.6. Products of Reviews
Detailed review findings and recommendations are
documented and provided to the project team, either by annotating
individual life cycle products and/or in a separate report.
Summary findings and recommendations are documented in a
memorandum attached to the Decision Paper submitted to the
approval authority. If there are significant unresolved issues
regarding the findings or recommendation of the review, the
reviewer clearly notes these issues.
3.3. Approvals
3.3.1. Purposes of Approval
Approvals have four major purposes:
o To confirm and document program management commitment
to the project and its management approach;
o To obtain program management acceptance of the project
activity results, recommendations, and products of the
current life cycle stage;
o To approve continuation with the next stage of the
life cycle; and
o To confirm the continued commitment of resources to
the project for the remainder of the life cycle.
3.3.2. Identification of Needed Approvals
Approvals are provided in all stages. Each approval to
be obtained is identified in the Project Management Plan, with
the timing, focus, and approval authority noted for each
approval.
3.3.3. Approval Authority
The approval authority is determined by the values of the
approval criteria (see Section 4), and is either an Office
Director or the OSWER Information Management Steering Committee.
Depending on the specific requirements of a particular project,
the approval level may be changed or augmented as follows:
-------
OSWER DIRECTIVE *SC2a.GOa
o Office Directors may re-delegate approval within their
own offices, or may elect to elevate it to the OSWER
Information Management Steering Committee.
o The SIRMO has the authority to elevate approval to the
OSWER Information Management Steering Committee.
The approval authority is identified in the Project
Management Plan.
3.3.4. Conduct of Approvals
The Project Manager and reviewer will agree on the method
for conducting approval presentations. The approval authority
may require a formal presentation 'of some or all of the life
cycle products and/or the results of the review(s), or may
examine and approve the products without such a presentation.
3.3.5. Timing of Approvals
Approval should be timely upon completion of the required
reviews.
While awaiting an approval, a project may continue with
the subsequent stage (except when progressing from Implementation
to Production). If the project completes that stage while the
earlier approval is still pending, work stops until the pending
approval is received.
The project may continue beyond the next stage if so
directed in writing:
o by the IMC, for projects subject to approval by an
Office Director, or
o by the OSWER Senior IRM Official (SIRMO), for projects
subject to approval by the OSWER Information
Management Steering Committee.
3.3.6. Products of Approvals
Approval of a specific phase or stage is documented by a
formal memo, or by signature on the Decision Paper.
An approval may provide specific direction to the project
team, and modify the project approach, functional and data
requirements, and/or system design presented in the Decision
Paper.
-------
CSWER OlfiCTlVs i. £-C£2.C2;-
4. THRESHOLD ANALYSIS
4.1. Purpose of Threshold Analysis
The purpose of threshold analysis is to determine the
appropriate project review and approval levels.
4.2. Performing Threshold Analysis
Exhibit 4-1 lists the criteria used in determining
project review and approval levels/ and their values.
There are two project review and approval levels:
Level I Level II
Review Information Management Information
Coordinator(s) (IMC(s)) Management
and Coordinator
Senior Information
Resources Management
Officer (SIRMO)
Level I Level II
Approval Designated Office . Office Director
Director(s) and OSWER ' .
Information Management
Steering Committee
To determine the appropriate review and approval levels
for a given project, the Project Manager determines the project's
values for the criteria listed in Exhibit 4-1. Any project with
at least one Level I value is a Level I project. Exhibit 4-2
depicts the process of threshold analysis.
Threshold analysis is repeated in each phase or stage
from Initiation" through Implementation, and documented in the
appropriate System Decision Paper. ' Repeating the analysis
ensures that a proper level of review and approval is applied to
any significant changes to the requirements or design of the
system. If the appropriate review or approval level changes as
the result of the analysis, the change is documented in the
Project Management Plan and noted in the System Decision Paper.
8
-------
OSWEH DIRECTIVE i«C2S.C3a
EXHIBIT 4-1
CRITERIA FOR REVIEWS AND APPROVALS
Criterion
Cost
Level I Value(s)
Cost exceeds the
available budget
support of the.
sponsoring OSWER
office/ or
Investment cost, annual
operating cost, or
total life cycle cost
meets OMB reporting
requirements*
Level II Value(s)
Sponsoring OSWER office
can provide sufficient
funding with available
budget support, and all
costs are below OMB
reporting requirements*
Organizational
.Impact/People
Requirements
[for initial
creation of the
system, ongoing
operation after
implementation,
or both]
Requires FTE support
from multiple OSWER
program offices, or
Requires FTE support
from Regional offices,
or
Requires FTE support
.from state agencies
FTE support will be
required of only a
single OSWER program
office; no FTE support
required from other
OSWER offices, Regional
offices, or state
agencies
Data Sharing
Data are created/
collected/ and/or used
by multiple OSWER
program offices/
Regional offices/
and/or state agencies
Data are created/
collected, and used
within a single OSWER
program office; no data
are created by or.
collected from Regional
offices or state
agencies
Software/
Technology
Sharing
Under consideration as
a potential OSWER
standard, or
Deviates from existing
Agency standards
Not a candidate OSWER
standard, and fully
adheres to applicable
standards
- OMB reporting requirements are investment cost (Initiation
phase through Implementation stage) greater than or equal to
$5 million, or annual support costs greater than or equal to
$1 million.
-------
EXHIBIT 4-2: THRESHOLD ANALYSIS PROCESS
Cost
Exceeds
Sponsoring
Office's Budget or
Meets OMB
Reporting
Req't*
Requires
FTE Support
From Multiple
OSWER Offices,
egions, and/
States
Data
hared Across
Multiple OSWER
Offices, Regions,
and/or States
Potential
OSWER \NO
Standard Software
or Deviates from
Agency
tandard
Requires Level I
Review and Approval
Requires Level II
Review and Approval
* Investment costs (Initiation phase through
Implementation stage) equal or exceed $5 million,
or annual support costs equal or exceed $1 million.
REVIEW AND APPROVAL THRESHOLDS
Level I
Review IMC(s) and OSWER SIRMO
Approval Designated Office Directors)
and IM Steering Committee
Level II
IMC
Office Director
-------
OSWER DIRECTIVE »aQ2Q.QOa
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
Part 3: Practice Paper
Project Participation
and Coordination
January, 1989
-------
OSWER DIRECTIVE «90aS.QQa
TABLE OF CONTENTS
Page
1. Practice Paper Purpose 1
2. Roles and Guidelines for Selecting Participants 2
2.1 What is Meant by Project Participant 2
2.2 Program Management 6
'2.3 Program Staff 6
2.4 Project Management 8
2.5 Project Staff 8
2.6 Quality Assurance 10
2.7 Procurement 11
2.8 Timing of Role Designation 13
3. Other Factors for Selecting Project Participants 13
EXHIBITS
1-1. Illustration of Roles and Responsibilities of
Concept Phase 3
1-2. Roles and Responsibilities Four Major Groups 4
2-1. Multiple Roles for System Users 7
-------
PROJECT PARTICIPATION
-------
OSWER DIRECTIVE #9023.003
1. PRACTICE PAPER PURPOSE
This practice paper constitutes a section of Part 3 of the
Office of Solid Waste and Emergency Response (OSWER) System Life
Cycle Management Guidance. It describes the typical roles to be
filledin conducting an information system project, and provides
suggestions for determining the organizations that should fulfill
each role. It expands on the roles identified in Parts 1 and 2
of OSWER's System Life Cycle Management Guidance. This paper
also suggests organizations that should participate in the
project/ or with whom the project should be coordinated.
The topics addressed in this practice paper include:
o Common roles to which specific organizations and
individuals are assigned to accomplish the information
system project;
o Relationships between project roles;
o Organizations that should be considered for
participation or coordination in an OSWER system
project; and
o Criteria to be used, in determining the appropriate
. organizations from which individuals should be assigned
to 'each role.
Proper attention ' to project participation serves several
important purposes in support of the system life cycle:
o Helps ensure that the system focuses clearly on the
information management problem and related requirements
by identifying the appropriate organizations and
enabling the most knowledgeable individuals to
participate actively in the project;
o Helps ensure that project support continues throughout
.the entire life cycle with the involvement of
appropriate program managers and likely system users;
o Helps ensure that the system is designed well and
implemented effectively by ensuring that organizations
and individuals with the proper technical skills and
real experience are involved in the project;
o Helps provide clear expectations of the participants in
a system project, particularly of novice participants,
regarding their respective roles and responsibilities on
the project.
This practice paper discusses project participation from a
broad perspective, considering the participation of offices
-------
within OSWER, other Agency offices, and the external
organizations (such as State agencies) that implement OSWER
programs and are likely users of OSWER systems. It also
considers organizations within the Agency that may be called on
to provide expertise regarding information management analysis,
technologies, and procurement.
It should be noted that the selection of project
participants and the overall organization of the project team are
important parts of the Project Management Plan and are documented
in the Project Management Plan. A separate practice paper is
available that provides more information about the project
management plan.
This practice paper on Project Participation and
Coordination does not address*the responsibilities of individual
project participants with respect to the individual activities of
the system life cycle. Rather, it describes the general nature
of the responsibilities for the organizations assigned specific
project roles. Part 2 of the System Life Cycle Management
Guidance identifies the specific activities of the life cycle,
and the responsibilities for accomplishing them. Exhibit 1-1
provides an example, taken from Part 2 of the Guidance,
illustrating specific responsibilities for several of the
activities of a single phase, the Concept phase. Exhibit 1-2
illustrates how the roles identified in Part 2 can be viewed more
simply as four groups:
o Program Group
o Project Team
o Quality Assurance Group
o Procurement Support Group
This practice paper does not describe how specific
individuals should be selected to participate in the project, nor
does it focus on the determination of potential contractor
support needs. Assistance In these matters can be obtained from
the OSWER Information Management Staff (within the immediate
office of the Assistant Administrator), OSWER Information
Management Coordinators (IMCs), and the Agency Office of
Information Resources Management (OIRM).
2. ROLES AMD GUIDELINES FOR SELECTING PARTICIPANTS
2.1. What is Meant by 'Project Participant'?
The life cycle management process describes a broad and
diverse set of activities for solving an information management
problem. The activities of the life cycle range from defining
the problem and selecting the solution, to developing, operating
and maintaining the solution. Project participants are those
organizations and individuals that contribute in some substantive
-------
EXHIBIT 1-1
ILLUSTRATION OP ROLES AND RESPONSIBILITIES
OF CONCEPT PHASE
CONCEPT PHASE
ROLES AND RESPONSIBILITIES
ROLES AND RESPONSIBILITIES
PROJECT QUALITY
OSWER PROGRAM
KUJC.V.1 vur._...
STAFF ASSURANCE PROCUREMENT
OSWER PROGRAM
MANAGEMENT
ACTIVITIES
DEFINE AND DOCUMENT
HIGH-LEVEL FUNCTIONAL
REQUIREMENTS DEFINITION
DEFINE AND DOCUMENT
HIGH-LEVEL DATA
REQUIREMENTS DEFINITION
DOCUMENT CONNECTIONS
BETWEEN HIGH-LEVEL
FUNCTIONAL AND DATA
REQUIREMENTS
ASSESS CAPABILITIES OF
CURRENT SYSTEMS, DATA
BASES, AND PROCEDURES
DEFINE ALTERNATIVE
CONCEPTS
EVALUATE CONCEPTS
AGAINST REQUIREMENTS
SUPPORT
SUPPORT
SUPPORT
SUPPORT
SUPPORT
SUPPORT
LEAD PERFORM REVIEW
LEAD PERFORM REVIEW
LEAD PERFORM REVIEW
LEAD PERFORM REVIEW
LEAD PERFORM REVIEW SUPPORT
* ,
LEAD PERFORM REVIEW
CO
CD
8
-------
EXHIBIT 1-2
ROLES AND RESPONSIBILITIES FOUR MAJOR GROUPS
CONCEPT PHASE
ROLES AND RESPONSIBILITIES
ACTIVITIES
DEFINE AMD DOCUMENT
HIGH-LEVEL FUNCTIONAL
REQUIREMENTS DEFINITION
DEFINE AND DOCUMENT
HIGH-LEVEL DATA
REQUIREMENTS DEFINITION
DOCUMENT CONNECTIONS
BETWEEN HIGH-LEVEL
FUNCTIONAL AND DATA
REQUIREMENTS
ASSESS CAPABILITIES OF
CURRENT SYSTEMS, DATA
BASES, AND PROCEDURES
DEFINE ALTERNATIVE
CONCEPTS
EVALUATE CONCEPTS
AGAINST REQUIREMENTS
ROLES AND RESPONSIBILITIES
DSWER PROGRAM OSWER PROGRAM
MANAGEMENT STAFF
SUPPORT
SUPPORT
1
SUPPORT
. SUPPORT
SUPPORT
.
SUPPORT
PROJECT PROJECT
1ANAGEMENT STAFF
LEAD PERFORM
LEAD PERFORM
LEAD PERFORM
LEAD PERFORM
LEAD PERFORM
*
LEAD PERFORM
QUALITY
ASSURANCE
REVIEW
REVIEW
REVIEW
REVIEW
REVIEW
REVIEW
PROCUREMENT
SUPPORT
CD
-------
OSWER DIRECTIVE #9028.QOa
way to the evolution of the solution, which often takes the form
of an automated information system and/or data base.
All project participants need not be experts in ADP or
information management technology. Although having participants
with this expertise is vital, other participants may be selected
due to their knowledge of the waste management program and/or
their project management skills.
The selection of project participants depends heavily on
the nature of the information management problem, the scope of
organizations that experience the problem, and a number of
characteristics of the solution to the problem, such as the
specific data to be collected and the hardware to be used (for an
automated system).
This section of the practice paper describes the roles of a
typical system project and provides general guidelines for .
identifying the candidate organizations that should be called on
to participate in a system life, cycle project. Section 3
identifies specific characteristics of information management
.problems and their solutions that will help in choosing the
appropriate organizations.
The specific roles described in this section are the roles
shown in Exhibit 1-1:
o Program Management,
o Program Staff,
o Project Management,
o Project Staff,
o Quality Assurance, and
o Procurement Support.
Project participants may serve in a number of different
roles for a given project. Some will be members of the project
team, either in the role of project management or project staff.
Others will be involved in the project to a lesser degree, but
still will have an important role. In selecting the
participants, it is critical that all participants, their
respective supervisors, and the system project manager have a
clear and consistent understanding of the participants' extent of
involvement in the project (e.g., full time, half time, one half
day per week). This involvement should balance the needs and
priority of the project with the other program responsibilities
the participants will continue to carry.
It is important to note that the users of the system serve
in three different roles. Some users will be members of the
-------
uawtH uiHtuuvt
project team. Others will be members of a Quality Assurance
Group. The remaining users may contribute to the project by
assisting the project team or the Quality Assurance Group in a
less formal capacity. This variety of user roles is illustrated
in Exhibit 2-1.
2.2. Program Management
Program Management sponsors and funds the project. Key
responsibilities include identification of the information
management problem, selection of the project manager and other
members of the project team, and approval of the recommended
decisions and products of the project team throughout the life
cycle.
The program management role usually is assumed by those
organizations most directly affected by the information
management problem. In general, the designated program manager
for an information system project will be a member of the
organization that will serve as the- primary user of the data to
be processed by the system. However, some projects have broad
organizational impact for these projects, the program
management role must be shared by multiple organizations.
Candidate organizations for this role include regional as
well as headquarters organizations.
2.3. Program Staff
Program staff represent the day-to-day users of the system,
and provide a very strong programmatic perspective of the
information management problem and viability of the solution.
Some members of the program staff will be assigned to the project
team. Other members will have important responsibilities such as
providing input and advice to the project team regarding the
functional and data requirements, reviewing suggested system
alternatives and the system design, and participating actively in
acceptance testing of the system before it is made available to
users.
Similar to the program management role for a project, 'the
designation of participants serving as program staff reflects
those organizations most directly affected by the information
management problem. The designation of program staff must
recognize those organizations that collect and/or generate the
data needed to solve the problem, as well as those organizations
that use the data.
For problems with broad organizational impact, individuals
may be designated from multiple organizations. Widespread
participation in this role is common for large systems, and could
involve representatives of multiple OSWER offices at
headquarters, other program offices, the regional offices, and
-------
OSWEB DIRECTIVE i?9028.0Qa
EXHIBIT 2-1:
MULTIPLE ROLES FOR SYSTEM USERS
PROJECT
TEAM
Users in an
Assistance Role
A
PROGRAM
MANAGEMENT
PROGRAM
STAFF
/
/
EXTERNAL
fcNIZATIO
local agencies*
QUALITY
ASSURANCE
GROUP
= SYSTEM USERS
-------
UOWtK UlHffiHVE 09028.008
state agencies and agency associations. In some instances it may
be appropriate to involve industry as well.
2.4. Project Management
The project management role provides direction and
management of the system project, and retains ultimate
responsibility and accountability for solving the information
management problem within the approved timeframe and budget.
Each system project has a single Project Manager, an
individual drawn most often from the organization(s) sponsoring
the project. The project manager may be drawn from a regional or
headquarters organization.
For a very large system project, such as for a national
system involving several OSWER offices and the regions, the
project management role may be performed by group of individuals.
The establishment of a project management group helps ensure that
a broad programmatic and organizational perspective is applied to
key analyses and to recommendations to program management
throughout the life cycle. For these projects, a single Project
Manager is designated to manage the day-to-day work of the
project, and is a member of the project management group
(referred to by a name of its choosing). The relationship of the
Project Manager to this group is established on a case by case
basis by the program manager. For some projects, this group will
be advisory to the Project Manager. For others, this group will
have decision authority and will direct the Project Manager.
The selection of the organizations) and individual(s) for
the project management role is critical to the success of the
project. This selection should carefully consider, and balance,
the following factors:
o Knowledge of program policy and operations commensurate
with the scope of the information management problem;
o General project management ability, including management
.of internal personnel and contractors (if appropriate);
o Information system project management ability;
o Expertise in information management methods, tools and
technologies; and
o Expertise in conducting procurements of information
management technology or services (if applicable).
2.5. Project Staff
Project staff perform the majority of project activities,
working under the direction of the Project Manager. These
participants provide the full range of programmatic and technical
8
-------
OSWEfl DIRECTIVE #902B.OOa
knowledge, skills, and abilities needed to accomplish the
project, using individuals drawn from programmatic and
information management organizations as appropriate. Quite often
contractors serve as members of the project staff. The
organizations and individuals filling this role, along with
Project Management, are commonly referred to collectively as the
'project team1.
The Project staff bring to the project team a combination
of the knowledge, skills and experience needed to successfully
analyze the problem, select an appropriate solution, and develop,
implement and maintain the solution until it is no longer needed.
The Project Staff brings together many areas of expertise:
programmatic, systems analysis, information management
technology, and other areas.
The organizations and individuals selected for the Project
Staff should provide the needed abilities, and represent,
collectively, the organizations who are experiencing the
information management problem or will be involved in developing,
operating, and/or maintaining its solution. Project teams should
always include representatives of user organizations, including
organizations that will use the information provided by the
system, and also organizations that will generate and/or collect
the data to be processed by the system. Thus, the project staff
often includes individuals outside of OSWER for a headquarters-
based project, and individuals, outside of the regional Waste
Management Division for a single-region project.
The selection of the Project Staff is usually a joint
effort .of the Project Management and Program Management roles.
Where coordination across program or organizational lines is
needed (e.g., selecting team members from Regional offices),
Program Management usually has lead responsibility for such
coordination to ensure that individuals are assigned at the
proper level and with the appropriate extent of commitment (i.e.,
level of effort) to the project.
For problems with broad organizational impact, individuals
may be .drawn from many organizations. Widespread participation
on the Project Staff is common for large systems. Listed below
are those organizations who may be called on to participate as
members of the project staff.
o Individual OSWER program offices,
o Regional Waste Management Divisions,
o Regional Environmental Services Divisions,
o Regional Enforcement Divisions,
o Regional Management Divisions,
-------
OSWER DIRECTIVE #9023.003
o Other Agency program offices,
o Office of Information Resources Management,
o OARM/RTP - National Data Processing Division,
t
o Other Federal agencies,
o State waste program management agencies,
o Associations of State agencies, and
o Industry and industry associations.
Section 3 of this practice paper provides additional
guidance to help in selecting the project staff.
2.6. Quality Assurance
The quality assurance role conducts informal and formal
reviews of system life cycle products, providing feedback to the
project team and advice to Program Management regarding project
approvals. The quality assurance role for a project is performed
by individuals representing programmatic and information
management organizations, with many of the same skills and
abilities of Project Management and Project Staff roles.
However, the individuals selected for quality assurance are not
members of the project team as defined above. The organizational
level for performing quality assurance, particularly for formal
project reviews, depends on the threshold level of the system.
(The OSWER formal review and approval process, including
threshold analysis, is explained in the practice paper on 'System
Life Cycle Reviews and Approvals'.)
i
Individuals assigned to the Quality Assurance role would be
drawn from many of the same organizations as the Project
Management and Project Staff. However, individuals assigned to
the quality assurance role cannot also be assigned to these other
roles. Doing so would result in a conflict of interest, with
these individuals ostensibly performing an "independent" review
of their own work, or of the work of their project team
associates. It should also be noted that although individuals
assigned to the Quality Assurance role, Project Management role
and Project Staff role are drawn from the same organizations, far
fewer individuals are assigned to the Quality Assurance role than
to the combination of Project Management and Project Staff.
Leadership of the quality assurance role (referred to as
'lead reviewer') varies, depending upo. the level of review and
approval, required -for the project. This level is determined by
.an analysis referred to as 'Threshold Analysis'. The OSWER
senior IRM official serves as lead reviewer for a Level I system,
and an Information Management Coordinator (at headquarters) or
another seasoned information management professional serves as
10
-------
OSWER DIRECTIVE 09028.00a
lead reviewer for a Level II system. The practice paper on
'System Life Cycle Reviews and Approvals' describes the
distinction between these two levels of system projects in more
detail.
It should be noted that some projects, particularly those
that are very large (over $1 million cost for Initiation phase
through Development and Implementation Phase) or will have major
impact on program operations, may at some point in their life
cycle be subject to a review or audit by the Agency Office of the
Inspector General (OIG). The interests and concerns of OIG may
have a major impact on the overall schedule and cost of
completing the system if not identified and resolved at the
proper point of the life cycle. Obtaining OIG participation (in
a Quality Assurance role) early in the life cycle would minimize
subsequent disruptions due to suggested changes and their
resolution. With early resolution, changes will have minimum
impact due to OSWER's ability to incorporate the changes in the
products of the current phase/stage, and by updating the Project
Management Plan for future phases and stages. Program Managers
are therefore strongly encouraged to request OIG participation in
these projects at their inception.
Should OIG decline to participate, the OSWER/AA Information
Management Staff should be informed. Several potential actions
will be considered: .
o Sending copies of approved system decision papers to
OIG, and
o Contacting OIG at the beginning of each of the following
phases and stages to participate in a Quality Assurance
role:
Concept phase
Definition Stage
Design stage
Implementation stage
Evaluation stage
The actiori(s) taken should be recorded in the Project Management
Plan.
2.7. Procurement
The Procurement role provides expert advice to the
Project Manager in planning for the acquisition of needed
resources other than EPA personnel. These resources include ADP
and communications hardware and software, and contractor support
services. The procurement role also participates actively in
(often leading) other procurement activities of individual system
projects.
11
-------
OSWER DIRECTIVE #902B.OOa
Although in many cases the needed resources can be obtained
via existing contracts, in some cases it will be necessary to
conduct a new procurement. Both of these methods of resource
acquisition require specialized knowledge, and conducting new
procurements (when necessary) requires exceptional expertise to
successfully navigate the intricacies of the procurement process.
As soon as a project identifies the need for resource
acquisition, it is essential that the Project Manager contact the
pertinent procurement support organization for additional
information and insight. For most projects/ the need to acquire
resources will be identified as part of the system concept. For
large projects in particular, the need to acquire contractor
support may be evident early in the Concept phase, or as early as
the Initiation phase.
Several organizations may provide procurement-related
support to the Project Manager. However, only the OARM
Procurements and Contracts Management Division (PCMD) can legally
commit the Agency to a new contract or modify the provisions or
terms of an existing contract. The other organizations listed
below can serve in an advisory capacity to the Project Manager.
In addition, some of them are required to approve procurement
requests before they are processed by PCMD.
Some projects will acquire different types of resources
and/or require the use of multiple contracts or procurements.
For these projects, individuals may be drawn from multiple
organizations, to handle multiple procurements. The specific
organizations to be contacted within the Agency for procurement
advice and support for a given project may include:
o OARM/Procurement and Contracts Management Division
t
o Regional Management Divisions, IRM Branch (for regional
projects), *
o OSWER program office Information Management Coordinators
(IMCs) (for headquarters projects), *
o OSWER/AA Information Management 'Staff, *
o Regional Waste Management Division IRM Planning
Coordinators (for regional projects), *
o Office of Information Resources Management (OIRM),
o OARM/RTP - National Data Processing Division, and
o Other Federal agencies with interagency agreements with
EPA, or applicable 'schedule1 contracts (e.g., General
Services Administration).
12
-------
OSWER DIRECTIVE #902B.OOa
Project Managers should contact one of the organizations
designated above with an asterisk (*) before contacting other
organizations for procurement support.
2.8. Timing of Role Designation
To ensure a successful solution to the information
management problem, it is important that all participants in the
project understand what their role is, how they fit into the
overall effort to solve the problem. Within the life cycle
process, the roles of Program Management and Project Management
are assigned during the Initiation phase. All other roles are
assigned as early as possible in the next phase of the life
cycle, the Concept phase. These assignments are refined, as
appropriate, throughout the remainder of the life cycle based on
the experience of the project.
3. OTHER FACTORS FOR SELECTING^ PROJECT PARTICIPANTS
The selection of specific organizations for the roles
described in Section 2 above should be sure to obtain the
participation of organizations with essential skills, insights
and experience. The listing presented on the following pages is.
intended to aid Project Managers and Program Managers in
identifying the appropriate organizations to participate in a
particular project.
13
-------
USWEH DIRECTIVE »JU28.QOs
FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
FACTORS PERTAINING TO THE INFORMATION MANAGEMENT PROBLEM
Information required from State,
County or Local level
organizations.
o OIRM - State/EPA Data
Management Project
o Regional Waste
Management Divisions
o Professional
associations such as
NGA, ASWPCA, ASTSWMO
Information required from EPA
Regional offices.
o Regional Waste
Management Divisions
o Regional Environmental
Services Divisions
o Regional Management
Divisions
o Regional Enforcement
organizations
Some of the required information is
classified/ sensitive or can be
claimed .as confidential business
information (CBI).
o OSWER/AA Information
Management Staff
o Office of Information
Resources Management
o Facilities Management
and Services Division
(Security Unit)
Some of the required information is
generated or collected by another
EPA program office (external to
OSWER).
Senior Information
Resources Management
Official (SIRMO) for
that program office
Office of Information
Resources Management/
Program Systems
Division
14
-------
QSWEfl DIRECTIVE #9028.0Qa
FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
Information will be collected
directly from State, County or
Local level organizations.
OIRM - State/EPA Data
Management Project
Regional Waste Management
Divisions
Information will be submitted by
EPA Regional offices.
o Regional Waste Management
Divisions
Any facet of system operation will
require FTE or contract resources
at the regional offices.
o Regional Waste Management
Divisions
o Regional Management
Division
Any facet of maintaining the
application software for the system
will be performed in the regions
(e.g., regional or state
enhancements, installation of
software upgrades/new releases from
headquarters)
o Regional IRM Branch
Part of the system will operate on
the EPA mainframe computers at the
NCC and/or WIC.
o OARM/RTP - National Data
Processing Division
National Computer
Center (NCC)
Washington
Information Center
(WIC)
Part of the system will use the EPA
telecommunications network
o OARM/RTP - National Data
Processing Division
Part of the system will operate on
the regional logical mainframe
computers
o OARM/RTP - National Data
Processing Division
o Regional IRM Branch
15
-------
FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
(Continued)
Part of the system will operate on
the regional minicomputers
o Regional IRM Branch
Part of the system requires the
installation of new computer
equipment (including personal
computers), or needs office
environmental conditioning
equipment.
o OIRM at headquarters;
Regional IRM Branch
o Facilities Division (at
headquarters or regional
office, as appropriate).
Part of the system requires the
installation of a local area
network (LAN), or the modification
of an existing (LAN) or any other
wiring (e.g., power supply,
telephone lines).
Part of the system will use shared
OSWER minicomputer capacity (e.g.,
Prime minicomputers).
o OIRM at headquarters;
Regional IRM Branch
o OARM/RTP - National Data
Processing Division
o Facilities Division (at
headquarters or regional
office, as appropriate).
o OSWER/AA Information
Management Staff
Part of the System will submit or
receive data from the Agencies
Financial Management System or
Document Control Register System
(including Payroll System).
o OIRM Administrative
Systems Division
o Agency Financial
Management Division
Part of the System will process
personnel data (potentially subject
to Privacy Act requirements)
o Office of Administration,
Personnel Management
Division
o OIRM Information
Management and Services
Division
Part of the System will use
software that is not an Agency
standard, as defined by NDPD.
o ^SWER Program Office
Information Management
Coordinator (for
headquarters)
o Regional IRM Branch
16
-------
FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
(Continued)
Part of the system, including
manual procedures, relates to
procedures for handling documents
or other artifacts that are
considered 'official record
material' (includes use of digital
imaging technology as well as
microfilm, microfiche, or other
hardcopy media).
OSWER/AA Information
Management Staff
OIRM Information
Management and Services
Division
Part of the system will use expert
system technology.
OSWER/AA Information
Management Staff
High volume printing of
documentation is necessary
Printing unit in
Facilities Division, OARM
(or regional counterpart)
OTHER CONSIDERATIONS
New information will be collected
by the Agency from sources outside
the Agency (OMB approval is
required)
Office of Policy,
Planning and Evaluation,
Office of Standards and
Regulations
17
-------
OSWER DIRECTIVE #9028.008
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
Part 3: Practice Paper
Expert Systems
January, 1989
-------
OBWER DIRECTIVE *902B.OOa
TABLE OF CONTENTS
Chapter 1 Expert Systems Defined
1.0 Introduction 1-1
1.1 Expert System Capabilities. 1-3
1.2 Expert System Benefits 1-4
1.3 Expert System Limitations 1-4
1.4 Differences Between Decision Support Systems,
and Management Information Systems '. 1-5
1.5 Expert System Components and Architecture 1-8
1.6 Application Types 1-9
1.7 Prototyping Overview 1-12
1.8 Sources of Additional Information 1-14
1.9 Quick Briefing on Expert Systems 1-15
Chapter 2 Implications of Using Expert Systems at EPA
2.1 Expert Systems for Policy Interpretation 2-1
2.2 Legal Issues ..2-1
2.3 Organization's Culture 2-4
2.4 Cross Cutting Considerations 2-6
2.5 Resource Requirements 2-11
Chapter 3 Initiation Phase
3.0 Introduction :. .3-1
3.1 Objectives 3-2
3.2 Decisions 3-2
3.3 Success Factors 3-3
3.4 Products 3-5
3.5 Activities 3-7
Chapter 4 Concept Phase
4.0 Introduction 4-1
4:1 Objectives 4-2
4.2 Decisions 4-2
4.3 Products 4-5
4.4 Success Factors 4-7
4.5 Activities 4-10
4.6 Knowledge Management 4-13
4.7 Feasibility Assessment 4-17
4.8 Knowledge Representation 4-17
4.9 Control Structures 4-19
4.10 The System Concept ..-. 4-20
4.11 Proof-of-Concept Prototype 4-20
4.12 Assigning a Project Team 4-20
-------
OBWER DIRECTIVE #9028.UUa
Chapter 5 Definition and Design Phase
5.0 Introduction 5-1
5.1 Obj ectives 5-1
5.2 Decisions 5-1
5.3 Products 1 5-2
5.4 Success Factors. 5-3
5.5 Selecting a Development Environment 5-5
5.6 Developer Qualifications 5-12
5.7 Design 5-13
5.8 Prototyping Steps 5-15
Chapter 6 Development Stage
6.0 Introduction 6-1
6.1 Objectives 6-1
6.2 Decisions 6-1
6.3 Products 6-1
6.4 Success Factors .'.6-3
6.5 Activities 6-5
6.6 Knowledge Acquisition 6r6
6.7 Collection Methods 6-7
6.8 Resolving Conflicts. 6-10
6.9 Knowledge Notebook 6-11
6.10 Prototyping Steps 6-11
6.11 Testing and Validation 6-12
Chapter 7 Implementation Stage
7.0 Introduction 7-1
7.1 Obj ectives 7-1
7.2 Decisions 7-1
7.3 Products 7-2
7.4 Success Factors ; 7-3
7.5 Activities 7-5
7.6 Distribution Strategy 7-6
Chapter 8 Operation Phase
8.0 Introduction 8-1
8.1 Objectives 8-1
8.2 Decisions 8-2
8.3 Success Factors 8-3
8.4 Products 8-4
8.5 Activities 8-6
Appendix A Evaluating Expert System Shells A-l
Appendix B Glossary B-l
-------
OSUER Directive #9028.OOa
CHAPTER 1
EXPERT SYSTEMS DEFINED
1.0 INTRODUCTION
The purpose of this practice paper is to help software developers
and project managers in EPA's Office of Solid Waste & Emergency Response
(OSWER) understand the issues involved in successfully applying expert systems
technology to OSVER information management problems.
This paper explains when and where expert systems can be used and
how the development of expert systems should be managed in OSWER. It is an
extension of the OSWER System Life Cycle Management. The paper focuses on the
expert system life cycle as it relates to the conventional life cycle phases.
For this Practice Paper to be used effectively, the project leader and
developer should have a firm understanding of the OSWER System Life Cycle
Management Guidance, or LCM Guidance as it will be called elsewhere in this
document. Further, this paper does not explicitly define or describe in
detail how to build an expert system. The reader is responsible for obtaining
this information through formal training, seminars, computer literature (see
the bibliography in Section 1.8), or other media. The end of CHATTER 1 has a
Quick Briefing on Expert Systems.
The paper is divided into the eight chapters. Those specifically
related to the LCM Guidance (CHAPTERS 3 through 8) should be read in sequence.
When reading the chapters, bold-face terms are products associated with each
of the LCM phases. The first occurrence of terms defined in the glossary are
bracketed like this [term]. Each chapter has several additional features in
common with the rest:
o An introduction;
o A statement of objectives;
o A list of the decisions that must be made in the life
cycle phase;
o A list of success factors; and
o A description of activities that are to be conducted
during each phase or stage.
Each chapter contains an illustration of the objectives, decisions,
and products for the phase. The illustration at the start of each chapter
provide the reader with an orientation point in each chapter.
1-1
-------
OSWER Directive #9028.OOa
The chapter you are reading, Expert Systems Defined, introduces the
concept of [expert systems]. It explains the capabilities, benefits, and
limitations that may be associated with their use. It provides examples of
typical expert system applications.
CHAPTER 2; implications of Using Expert Systems, gives an overview
of the possible impacts expert systems could have on OSWER. It identifies
possible legal and liability issues as well as potential impacts on Agency
policies. The chapter describes the cross-cutting project management issues
that are addressed in multiple phases of the system life cycle. These are
project management plan, project participation, reviews and quality assurance,
project approvals, configuration management, data administration,
methodologies and [tools], cost-benefit analyses, and [knowledge management].
Finally, the chapter addresses the resources required throughout the expert
system life cycle.
CHAPTER 3; Initiation Phase, describes the tasks involved in problem
definition and in determining the need for an automated solution. This
chapter explores characteristics of a problem that suggest a possible expert
system solution.
CHAPTER 4; Concept Phase, depicts the identification of a feasible,
timely, cost-effective solution to the problem. This chapter contains
information on the use of proof-of-concept prototyping to refine the
solution, [knowledge representation], and management techniques, expert system
[control structures], and system development justification approaches. Also,
it specifies required qualifications for system developers.
. CHAPTER 5; Definition and Design Phase, discusses issues as they
relate to confirming the suitability of the [System Concept] and determining
detailed functional requirements. -It discusses all aspects of selecting a
development environment including [knowledge base] creation, migration to
delivery environments, and user interfaces.
CHAPTER 6; Development Stage, describes issues in developing the
expert system. This chapter introduces various [knowledge acquisition]
methodologies, sources, and [conflict resolution] techniques. Also, it
defines the means and importance of testing and validating the system.
CHAPTER 7, Implementation Stage, identifies the strategies for
distributing expert systems. It includes the issues of beta testing, user
registration, hardware and software requirements, training, licensing,
documentation, configuration management, and version control.
CHAPTER 8, Operation Phase, focuses on the Production, Evaluation,
and Archive Stages of the life cycle. It describes expert system maintenance,
end-user support requirements, knowledge revalidation options and maintenance,
ongoing training and documentation, and software updates.
1-2
-------
OSUER Directive #9028.OOa
The paper concludes with two appendices. Appendix A describes the
process of evaluating [rule-based] expert system software packages (e.g.,
[shells]) and provides an evaluation form. This appendix will be useful to
project managers and software developers during the [Definition and Design
Phase] when the development environment may be a rule-based shell. The
process of evaluating shells using other knowledge representation schemes
(e.g., [frames], object-oriented programming, etc.) has not been defined.
This type of information is widely available in the professional literature.
See Section 1.7 for references on further reading. Appendix B is a glossary
that provides a definition of the [artificial intelligence] terms and concepts
used throughout this practice paper.
1.1 EXPERT SYSTEM CAPABILITIES
In order to understand how expert systems can be used
advantageously, the project manager or software developer must first
understand how knowledge processing in expert systems differs from
conventional data processing. Expert systems are unique in their ability to
process [knowledge], not just data. Knowledge processing differs from data
processing in the type of information used, the techniques used to analyze the
information, and in the form that the results of the knowledge processing are
presented to the [user].
Conventional systems limit the developer to data representation
using only numbers and.text. They process data using complex [algorithms]
that complete a discrete number of steps to reach a predetermined conclusion.
Expert systems permit knowledge representation -- the encoding of human
decision-making processes using symbolic terms or [symbols]. Because expert
systems process knowledge, they are often referred to as [knowledge-based
systems].
The ability to represent knowledge in symbolic terms expands the
range of analysis techniques that computers can apply to information thus
enabling a system to emulate some aspects of human performance. The expert
system uses problem solving procedures such as pattern-matching to reason
about the symbolic terms. An example of [symbolic reasoning] by an expert
system that determines the daily forecast is, "if the sky is cloudy, then the
forecast might include rain." This phrase could be used within an expert
system. A conventional system would require that the symbolic terms such as
"sky" and "cloudy" be defined concretely as numbers or text. The expert
system uses pattern-matching on the phrases "sky is cloudy" and "forecast
might include rain" to reach a conclusion about the forecast without requiring
definition of each of the terms within the phrases. A conventional system
would require that the developer define a set number of steps to,determine the
daily forecast. The expert system examines all possible .solutions using the
problem solving procedure that has been programmed into it.
1-3
-------
OSWER Directive #9028.OOa
The combination of problem solving procedures chat are built into
expert systems, together with the developer's ability to define problems using
symbolic terms, gives expert systems the capability to store and manipulate
more complex relationships between individual pieces and groups of information
than can be accomplished with the processing supported by conventional
systems.
In addition to knowledge representation, expert systems also provide
the capability to simplify the user/computer interaction. Expert systems can
be programmed to employ more of the conventions that people use when
communicating with each other. Expert systems can be designed with the
ability to explain the "reasoning" used in reaching a recommendation and to
justify their approach to a problem, much as people do. The more
sophisticated expert systems employ a [natural language] processor to permit
users to consult with the system using near-English rather than structured
query languages or menus.
1.2 EXPERT SYSTEM BENEFITS
Expert system applications take advantage of the above capabilities
in two ways. First, information becomes more accessible so people can make
better decisions. Second, where useful information is accessible, people can
be more productive. OSWER can be more productive in the use of its personnel,
funding, and information resources through the application of expert systems.
The two major benefits associated with expert systems are better decision
making and increased productivity. These benefits are described below.
1.2.1 Better Decision Making
Expert systems improve the quality of decision making by providing a
mechanism for pooling the.knowledge of multiple experts and making that
knowledge available to a. wider audience. This leveraging of knowledge results
in improved quality of complex work products such as permits, technical
reports, and analyses that recommend actions.
Expert systems establish a basis for defensible decision-making by
capturing and applying knowledge in verifiable form. For example, in
developing work products such as management reports and environmental
analyses, a given set of inputs, no matter how complex, should result in
consistent results given closely similar data, advice, or recommendations. In
addition, the process of reaching a conclusion can be explicitly demonstrated.
This will ensure consistency in many decision-making activities.
1.2.2 Increased Productivity
Expert systems offer significant, measurable increases in
productivity by effectively capturing the knowledge of experts and by
minimizing the loss of [expertise] and knowledge due to attrition. Expert
1-4
-------
OSUER Directive #9028.OOa
systems provide a means of extracting, storing, and sharing knowledge in a
variety of disciplines. Thus, more people have access to expertise. In turn,
the experts are freed from relatively mundane tasks to focus on demanding
ones.
1.3 EXPERT SYSTEM LIMITATIONS
Expert systems provide valuable new capabilities, but they also have
clear limitations. As with all new technology, developers must weigh the
limitations associated with the use of expert systems technology against
benefits. Because expert systems emulate human performance in decision
making, they may be incorrectly thought of as having the capacity to make
independent judgments. Expert systems are capable of communicating advice
that has been coded into them. They are not capable of producing independent
decisions. Their application is limited to strictly defined domains (i.e.,
areas of expertise where boundaries on what .expertise should be included in
the system can be defined); their performance degrades dramatically when
dealing with information that is beyond those boundaries.
Expert systems can manipulate only symbolic information, that is,
all "real" information that is collected by observing an event. For instance,
temperature and humidity in the case of weather forecasting must be translated
into a form acceptable to the expert system. Any errors and biases
incorporated in the translation process will be accepted by the expert system
without question. An example of a translation bias is if temperature
measurements input to the system are in Fahrenheit when the logic or knowledge
encoded into the system is based on Celsius measurements, then the conclusions
reached will be invalid.
Another limitation is the difficulty developers might experience
when linking expert systems to conventional data processing systems. This
would involve building a hybrid or embedded system. New expert system
products are appearing regularly to bridge the gap between expert and
conventional systems. However, until a greater number of these products
exist, software developers will experience difficulty making expert system
software communicate with conventional software.
1.4 DIFFERENCES BETWEEN DECISION SUPPORT SYSTEMS,
MANAGEMENT INFORMATION SYSTEMS, AND EXPERT SYSTEMS
Based on an understanding of the capabilities, benefits, and
limitations of an expert system, the project manager or software developer
needs to consider the differences between expert systems and conventional
systems such as database management systems. The following discussion will
provide background needed to determine whether expert systems are the
technology of choice for an application under consideration.
1-5
-------
OSUER Directive *9028.00a
All software tools and applications bear fundamental structural
similarities to each other; their differences lie in the capabilities
associated with each and in the methods or mechanisms used to achieve those
capabilities. Expert systems can be distinguished from conventional systems
based on those differences. For example, conventional systems do not have the
capability to handle uncertainty. They are developed using high level
programming languages such as COBOL or Fortran, and these languages must
explain in exacting detail how to solve a given problem. They provide no
capacity for vagueness or uncertainty.
Expert systems accommodate uncertainty through weighting schemes.
These weighting schemes calculate certainty levels by determining if a piece
of information is correct or valid and increment the level of certainty
accordingly. The inability to process uncertain data* confines the power of
conventional systems to linear programming tasks, that is, tasks where steps
are performed in sequence regardless of the inputs, and which are number- or
text-intensive. While conventional systems are developed using conventional
languages, expert systems are often developed with software referred to as
"shells." A shell provides a framework to build expert systems. Appendix A
provides a framework for evaluating shells.
Expert systems are also developed using LISP or [PROLOG]. These are
the recognized computer languages of artificial intelligence (AI). These
languages simplify the expert system development process because they are
interactive, and they provide feedback to the programmer as the program is
being written. This capability allows the programmer to see results more
quickly. Additionally, LISP and PROLOG code is generally more like English
and can be understood more readily than can the standard code of high level
languages. AI languages give the programmer the added capability of more
flexibility. Whereas LISP and PROLOG permit programmers to tell the computer
what to do, conventional languages require them to tell the computer how to do
it. With LISP and PROLOG', less development time is spent defining each piece
of information that might enter the system. The distinction between use of
programming languages and expert system shells is that the [inference engine]
in the shell has already been validated by the vendor. Shells are good places
to start prototyping work, and the developer can move to a programming
language if the complexity of the task warrants building the inference engine
from the ground up.
There are additional differences between expert and conventional
systems. These are described below.
1.4.1 Decision Support Systems
Decision support systems (DSS) allow users to combine their judgment
with data to manipulate the data and produce reports. The users' judgement 5s
translated into an algorithm that is then programmed into the decision support
system using conventional programming languages such as COBOL or Fortran. 'The
algorithm typically gives the user flexibility to define data manipulation
1-6
-------
OSUER Directive #9028.00«
scenarios and report formats. A specialized interface allows the user to
easily define meaningful data manipulation scenarios without requiring an
understanding of the underlying data structures.
DSSs employ a variety of information management technologies to
solve structured or unstructured problems and support managerial judgment,
thus improving the effectiveness of decision-making. Typical DSS tools
include spreadsheet, geographic information systems, modeling packages, and
statistical packages. These systems are also linked to the functions of 4th
Generation" relational database management software.
Expert systems are a specialized subset of DSSs that employ symbolic
reasoning to manipulate data and produce reports. If the judgement employed
by the users is procedural in nature (i.e., requires a set number of steps to
reach a predetermined conclusion), then a conventional DSS should be
developed. However, if the user reaches a conclusion based on a variety of
factors that cannot be captured using an algorithm, then an expert system
could be developed.
1.4.2 Management Information Systems
Management information systems are computer-based information
systems that provide information to support management activities and
functions. They support four general types of activities: transaction
processing such as payroll, sales order, and inventory; operational control
which ensures that activities are carried out efficiently and effectively;
management control which measures performance, deciding on control actions,
formulating new decision [rules], and allocating resources); and, strategic
planning which involves developing strategy with which to meet organizational
objectives.
Executive information systems (EIS) are a specialized form of
management information system. They offer top level managers access to
corporate information summarized in a manner specified by the executives so
they get what they need in the required form and detail.
Management information systems do not typically have symbolic
reasoning capabilities. The graphic displays are static in the sense that
there is no [explanation] or interpretation facility. The user must know how
to format queries; the system cannot reformat queries if the user is not
sufficiently specific. Vague or uncertain information contained in a
management information system query may result in computer resource intensive
processing caused by extensive database [searches]).
1-7
-------
OSUER Directive #9028.OOa
1.5 EXPERT SYSTEM COMPONENTS AND ARCHITECTURE
Expert systems can be viewed as having three unique, isolated
components. See Section 1.8 for a graphic representation of the components of
an expert system. These are the user interface, knowledge base, and inference
engine. Most expert systems and system development tools have these
components. If project managers and software developers are aware of each
component during the Definition and Design Phase of the system life cycle, a
development environment can be selected that supports the components and level
of complexity desired for the system.
1.5.1 User Interface
The user interface permits the user and system to communicate. The
user interface may be simple. It can consist of questions posed to the user
or optional responses from which the user makes a selection by typing in the
response desired; or menus from which selections are made using the cursor
keys or a mouse. A sophisticated user interface may employ a natural language
processor written in a higher level language to ask and answer questions, to
explain or justify the system's conclusions, and to accept modifications to
the knowledge base.
1.5.2 Knowledge Base
The knowledge base contains the knowledge or expertise of the domain
which is the designated area of expertise for the system. In many rule-based
expert systems, the knowledge is encoded into sentences using an "If PREMISE,
then CONCLUSION" format. [Facts] may also be stored in the knowledge base
using a variety of formats such as STRONG_WIND - TORNADO. Rules are used to
encode knowledge that includes reasoning about .the inputs to the system, or
facts already encoded into the knowledge base. Facts are used to encode
passive knowledge that is concrete. For example, each entry in a table that
translates temperature readings from Celsius to Fahrenheit could be included
in a knowledge base as a fact.
1.5.3 Inference Engine
The inference engine contains the specific procedures and algorithms
for using the rules and facts in the knowledge base to solve a problem. One
possible procedure or strategy used by an inference engine in a rule-based
expert system is [backward chaining]. Using backward chaining, the inference
engine first considers a primary goal. The text of the goal is matched
against the conclusion. For example "if the sky is cloudy, then the forecast
might include rain". In this example "sky is cloudy" is a premise and "the
forecast might include rain" is a conclusion. Rule matching determines
whc:her that rule will contribute information to the resolution of the goal.
If the conclusion of a rule matches the goal, then the premises of the rule
are considered in turn. Each of the premises is considered to be an
.intermediate goal. Results of evaluating each goal are stored in memory to be
1-8
-------
OSUER Directive #9028.OOa
used when evaluating subsequent goals. This inference strategy is riot the
only inference strategy. Refer to Section 1.8 for sources of additional
information.
1.6 APPLICATION TYPES
The ability of expert systems to reason about rules as well as input
data makes them particularly adept at handling many different types of
problems. This Section describes the types of problems and resulting
applications that are well suited to the use of expert systems technology.
The generic "types" of expert systems discussed below are well documented in
the literature about expert systems.
1.6.1 Diagnosis and Classification
Diagnosis and classification expert systems select an answer from a
fixed set of alternatives on the basis of information input to the system
while it is reasoning. Much notable work has been done with expert systems in
the field of diagnostics, where problems with an object (animate or inanimate)
are diagnosed from observations. Medical examinations and electronic circuit
board analysis have been successfully emulated by this -type of system.
Diagnostic systems must be able to handle intermittent symptoms, causes of
faults hidden by non-related symptoms, and sometimes inconsistent models of
complex systems..
Stanford University's MYCIN has performed extremely well in
diagnosing bacterial infections in humans and 'recommending antibiotic therapy.
Another diagnosis and classification expert system, MUDMAN, diagnoses problems
with "mud" used in NL Baroid's oil well drilling and recommends new
compositions.
1.6.2 Data Analysis and Interpretation
Data analysis and interpretation systems select a hypothesis based
on measurement data and corollary information. They infer situation
descriptions from accumulated sensor data. Artificial vision, image analysis,
and surveillance all employ expert systems, although work is just beginning in
these areas. (Note: this guidance document does not address the more esoteric
research areas current in artificial intelligence.) Expert systems can deal
with incomplete or contradictory data. In addition, their explanation
mechanisms can demonstrate the reasoning that lies behind a complex
interpretation. Expert systems are often good at assisting managers in
managing complex and incompatible data sources in order to consistently reach
a known range of conclusions.
An example of such a system is DIPMETER ADVISOR, developed by
. Schlumberger, which analyzes data from oil well instruments. Stanford
University's DENDRAL analyzes molecular structures based on mass spectrogram
1-9
-------
OSUER Directive #9028.OOa
data. Another example of a data analysis and interpretation system,
PROSPECTOR, was developed at Stanford Research Institute to interpret data
about ore-grade deposit sites.
1.6.3 Design and Synthesis
Design and synthesis expert systems configure objects such as
computer systems on the basis of a set of alternate possibilities. The expert
system incorporates constraints that the system must meet as well as guidance
for steps the system must take to meet the user's objectives. Design expert
systems are used in configuring large mainframe computers, in designing
circuit boards, and in preparing large annual budgets. Expert systems permit
the exploration of the consequences of proposed design changes prior to
implementing them. They also facilitate the subdivision of large problems
into smaller ones and simplify coordination among the subsets.
Other examples include Honeywell's SYSCON configures their DPS 90
mainframe computers before assembly at the factory, and Digital Equipment
Corporation's XCON which designs large VAX computers. HI Class, developed by
Hughes Aircraft, solves circuit board assembly problems.
1.6.4 Prediction and Simulation
Prediction and simulation expert systems forecast what will happen
on the basis of current information .by depending on experience or employing
models or simulation. Situations involving prediction are well suited to
expert systems. Crop management and weather forecasting are two areas of
current interest. Predictor systems must be able to model the ways actions
change over time and to manipulate events that are ordered in time. They must
also be able to deal with incomplete information, generate multiple scenarios,
and use diverse 'data sources.
For example, the USDA's COMAX advises farmers on irrigation,
fertilization, and when to harvest. Purdue University's GRAIN MARKET ADVISOR
helps farmers determine the best way to market the grain they produce.
1.6.5 Monitoring
Monitoring expert systems obtain data on an ongoing situation
following its predicted or intended progress and alerting the user or system
if there is a departure from the expected or usual. Expert systems capable of
monitoring their environments compare observations to desired outcomes and
report discrepancies. They must recognize alarm conditions in real time and
avoid false reports of-problems or emergencies. Note that "real time" expert
systems are significantly different than those which allow the luxury of
reflection. Because of varying environental factors, monitoring systems must
vary their anticipation of alarm conditions with time and situation.
1-10
-------
OSWER Directive 09028.OOa
Air traffic control and nuclear power plant management are two
current fields of application. Additionally, NASA's NAVEX monitors controls
on space shuttle flights.
1.6.6 Instruction
Instruction expert systems are built depending on user needs to give
advice, furnish information, or perform various subtasks. Expert systems that
can serve as an on-line tutorial aid to students and diagnose areas of
deficiency involve the automation of instruction. These systems construct a
hypothetical description of a student's knowledge in order to interpret the
student's behavior. They identify remedial courses of action and develop
tutorial modes of communicating the remedial action to the student. They can
be used to tutor novices and to advance the development of human experts.
For example, DOPPLER DIAGNOSIS, developed by Dr. Evlin Kinney, helps
train doctors in the use of non-invasive echo effect equipment. CBT ADVISOR,
developed by Courseware, helps determine the suitability of instructional
units for computer based training.
1.6.7 Planning
Planning expert systems select a series of actions from a complex
set of alternatives to meet a user's goals. Because time and resource
constraints may not permit all goals to be met, the most desirable outcome is
sought. Planning systems are used in contingency environments. A planning
expert system carries out prepared plans of action and can be used in a time'
critical situations such as responding to a. natural disaster. Priorities must
be established in order to resolve conflicts among goals and subgoals may need
to be established to simplify complex interactions.
IBM's UNIT COMMITMENT ADVISOR assists in the shut down of power
plants. Oak Ridge National Laboratories has experimented with several shells
with hazardous spill containment.
1.6.8 Control
Control is a combination of monitoring a system and taking
appropriate actions. Control systems interpret, predict, repair, and monitor
system behavior. Thus, they incorporate many of the other problem areas
presented here. Problems addressed by control systems include battle
management, mission control, and business management.
Hitachi's TRAIN BRAKING ADVISOR regulates locomotive braking for
accuracy and comfort. COMPONENT IMPACT ANALYSIS SYSTEM, developed by Argonne
National Laboratories, advises nuclear power plant operators on valve and
switch settings.
1-11
-------
OSUER Directive #9028,OOa
1.6.9 Repair
Debugging and repair systems specify remedial plans and apply them
in limited areas. Computer programming is the most obvious area of
application, but medical diagnostic systems also have debugging aspects.
Expert systems could greatly enhance expertise in automotive, avionics,
network, and computer maintenance.
Troubleshooting Aid for F6502, developed by Tektronix, assists
technicians in the repair of F6502 instruments. Stone & Webster's PUMP PRO
advises on preventive maintenance of centrifugal pumps. Honeywell's PRESS
debugs operating systems software. MIT's Programmer's Apprentice assists in
automatic code generation and in isolation of logical inconsistencies in
programs.
1.7 PROTOTYPING OVERVIEW
Because prototyping is frequently used in expert system development,
the concept is introduced here and developed further in the relevant stages.
.Prototyping is inherently easier with expert systems than with
building most conventional programs. Building an expert system is an
incremental process; at any time during development, the partially built
system will function to a limited degree. The scope and capability of the
system will be limited to the depth and breadth of knowledge that has been
applied to the knowledge base. When the expert system encounters a case about
which it has not been given any knowledge, it simply will not offer advice or
it may offer nonsense advice. With most conventional programs during the
development stage of the conventional system's life cycle, if they start to
function and encounter an area where logic has not been coded,- they will
usually end without warning or explanation. Because partially completed
expert systems will run, developers can demonstrate the system as it is being
developed, thereby soliciting comments, suggestions, and feedback.
Expert system development lends itself better to a. detailed
prototyping cycle than to a long design period for several reasons. First,
experts prefer to analyze a working system rather than one on paper. Second,
users are more motivated when they see a working system. Third, experts can
better cite exceptions to rules that pertain to realistic problems.
Exceptions to rules tend to emerge as problems are encountered. There are
several categories of prototypes, each with its own role in expert system
development. They are described in Sections 1.7.1 through 1.7.4.
1.7.1 Proof-of-Concept-Prototype
The Proof-of-Concept Prototype is a small working system designed to
provide a preliminary feasibility assessment of the emerging solution before
additional resources are allocated. It is used primarily as a proof-of-
1-12
-------
OSWER Directive #9028.OOa
concept test to show inherent strengths and weaknesses of the concept. It is
most commonly built in the Concept Phase, and further refined in the
Definition and Design Phase. This type of prototype may consist of
"disposable code" or code that is used only to prove that an idea or concept
is feasible.
1.7.2 Demonstration Prototype
If the System Concept meets all requirements, the Demonstration
Prototype is constructed during the Definition and Design Phase. Its purposes
are to codify core knowledge and to verify the knowledge representation
schemes and control structures. The Demonstration Prototype is generally
small and specialized, based on a narrow subset of the overall problem domain.
If major revisions are needed either to the knowledge representation or
control strategies,, they are made at this point. Should a different approach
appear to be more workable, the Demonstration Prototype can be reevaluated.
1.7.3 Full Prototype
During the Definition and Design Phase, the Full System Prototype is
developed to encompass the entire problem domain. The purpose is to verify
the problem domain, the knowledge representation schemes and control
structures, data requirements, and interfaces. If the problem is properly
scoped, the Full Prototype will be of a manageable size. The prototype
, contains most of the core and supporting knowledge and the agreed-upon
knowledge representation and control structures. If major revisions are
needed to the knowledge base, the knowledge base, or the control structures, a
new approach can be designed following the guidelines in Chapter 5, Definition
and Design Phase.
Prototyping is an exploratory process. If at this point the design
is no longer valid, the prototype can be reevaluated based on the new
information and redesigned as appropriate. The efforts that went into the
development are not lost; they represent the design phase in conventional
systems.
1.7.4 Production System
Finally, the Production System is built during the Development
Stage. This is the actual system that will go out into the field, including
all user interfaces and links to external databases. After complete testing
and validation, any disclaimers about the use and liability of the expert
system are added to the system.
1-13
-------
OSUER Directive #9028.OOa
1.8 SOURCES OF ADDITIONAL INFORMATION
1.8.1 Books
Harmon, P. and King, D. Expert Systems: AI in Business, John
Wiley and Sons, New York, NY, 1985. This beginner's primer on expert systems
does not assume any background in computer science, or AI. Contains an
excellent bibliography.
Hayes-Roth, F., Waterman, D., and Lenat, D. Building Expert
Systems, Addison-Wesley, Reading, MA, 1983. Comprises a good description of
the [architecture] of an "ideal" expert system and compares and contrasts an
implementation in several early shells.
Waterman, D. A Guide to Expert Systems, Addison-Wesley, Reading,
MA, 1986. A well-written introduction to ES technology. Includes an
extensive bibliography of expert systems in various application areas.
1.8.2 Organizations. Journals, and Magazines
AI Expert, 500 Howard Street, San Francisco, CA 94105. This is a
popular magazine for people interested in AI. The articles tend to be in the
vein of BYTE magazine, although some deal with industry trends and
applications.
American Association of Artificial Intelligence (AAAI), 445 Burgess
Drive, Menlo Park, CA 94205. Publications include a directory of members, AI
Magazine, and proceedings of yearly conferences. The magazine offers a mix of
theoretical, practical, and general interest reading.
Computer Society of the Institute of Electrical and Electronic
Engineers (IEEE), 10662 Los Vaqueros Circle, Los Alamitos, CA 90720.
Publications include COMPUTER, IEEE Software, and IEEE EXPERT. IEEE EXPERT
magazine is devoted to articles on applied AI; articles on AI also appear -in
the other magazines.
1-14
-------
OSWER DIRECTIVE #9028.00a
1.9 QUICK BRIEFING ON EXPERT SYSTEMS
-------
What is an Expert System?
Perspectives
Definitions
Component parts
Examples
o
(02)
O)
a
-------
"Expert Systems11 Comprise a Branch
of "Artificial Intelligence"
AI is a collection of cutting-edge, integrated concepts and methods
from a variety of fields:
Decision Science
Computer Science
Linguistics
Psychology
~ Neurology
- Mechanical and Electrical Engineering
and many more...
AI is what enables computers to perform tasks that would
normally require an experienced or trained person
-------
"Ryert Systems" Are A Practical
of Artificial Intelligence
Expert systems have proven to be practical in areas such as:
Medical diagnosis;
~ Seismic analysis and oil prospecting;
Chemical analysis;
Preventive maintenance;
Crisis response; '
Sales and negotiations advice;
~ Games; and
Engine fault diagnosis.
The expert systems arena is one of the most commercially
advanced branches of AI.
CO
-------
Some Classic Definitions of
"Expert Systems"
11A computer system that emulates human expertise by making
deductions from given information using the rules of logical
inference."
11A system whose goal it is to perform convincingly as an
advisory consultant, exhibiting human expertise in a given domain
with self-explanation of reasoning on demand."
"A computer program that has built into it the knowledge and
capability that will allow it to operate at the expert's level. It will
usually be able to explain the lines of reasoning that led to their
decisions."
o
1
m
at
s
-------
Conventional vs. Expert Systems
Conventional
Unambiguous data
Strict formats
No mid-run explanations
Numeric processing
Major effort to modify
Expert
Uncertain/incomplete data
Rules of thumb
Mid-run explanations easy
Symbolic processing
Easier to modify
s
-------
Key Components of an Expert System
User
Depending on the system,
users can be technicians or
managers, and can be novices
or experts in the domain of
the expert system.
Deccribe current problem
& known facts; query system
for expert advice.
Interrogate the user; provide
interim and final expert advice;
explain underlying reasoning.
User Interface
The part of the system that allows the user to communicate with the system, typically using English-like sentences or
commands. The user interface also gives the user control over output types (e.g., graphics or tabular reports).
L
Knowledge
Base
A collection of rules of
thumb, fact tables, and
questions that embody the
procedures employed by an
expert and the knowledge used
by the expert to make decisions.
Inference
Engine
The part of the system which
performs deductive reasoning
based upon the contents of
the knowledge base and the
circumstances of the current
problem.
Programs
The software that drives
and coordinates the system,
and which performs "utility"
tasks, such as data base
maintenance.
-------
The Knowledge Engineering Process
Information is extracted from experts
Information (i.e., rules, facts, and procedures) is arranged in a
knowledge base
Information conflicts and voids are identified and resolved
o
ni
m
b
£
-------
Hardware Environment vs.
Expert System Application Characteristics
Environment
Microcomputer
Characteristics
Rapid/Proof-of-Concept
Prototypes
Script-based
Rule-based with
limited fact tables
Limited requirement
for access to external
data Files
Minicomputer
-or-
AI Workstation
> 1000 rules
> Processing speed a consideration
Resource-intensive processing
Efficiency of operation required
(i.e., AI architecture, or garbage collection)
Mainframe
Multi-user access
>1000 rules
Processing speed a consideration
Access Required to Large Data Bases
Resource-intensive processing
§
&
-------
OSUER Directive #9028.OOa
CHAPTER 2
IMPLICATIONS OF USING EXPERT SYSTEMS AT EPA
2.0 PURPOSE OF THIS CHAPTER
This chapter discusses some policy issues for expert systems.
2.1 EXPERT SYSTEMS FOR POLICY INTERPRETATION
Expert systems for policy interpretation are not cost-effective for
OSWER because of the dynamic nature of policy. The maintenance costs of
expert systems that address policy can be significant due to changing
directives from Congress and regulatory policies. Projects that address
policy should have sufficiently short payback to offset their risk of
obsolescence. Because of these difficulties, expert systems may be used to
cite guidance and regulations but policy interpretation should be avoided.
This does not impose restrictions on text retrieval systems -- such as
hypertext and smart indexing -- but interpreting the policy is to be left to
human decision makers.
2.2 LEGAL ISSUES
Legal issues for expert systems involve all matters of the law,
including liability. Legal issues include licensing and distribution rights
to the software, copyright infringement risks for software and documentation,
and access to information used in the expert systems.
2.2.1 Licens ing Agreements
All OSWER expert systems will adhere to the terms of negotiated
licensing agreements set up with software vendors.
2.2.2 Confidential Information
Expert systems and their contents are open to public inspection.
Therefore expert systems should not contain confidential business information
(CBI) or enforcement information, nor should they contain data covered by the
Privacy Act. Information used in the expert systems will be made available to
the public in accordance with the Freedom of Information Act.
2.2.3 Liability
Liability in the realm of expert systems has received much attention
from computer scientists, but has not been tested in the courts. Some
software liability cases have been tried -- such as the LOTUS 1-2-3
spreadsheet error suit -- but the field is relatively open to interpretation.
2-1
-------
OSUER Directive #9028.OOa
2.2.3.1 System Profile
Expert systems appear to be especially vulnerable to liability
issues because of their advisory nature in judgmental matters and
their high profile. The high profile can be addressed by
classifying the use of an expert system as an assistant or advisor.
This sets the user's expectations to a more reasonable level. If
people believe that they are receiving accurate, "expert"
information, then* they may act on it without the proper skepticism.
2.2.3.2 User Registration
Another means of reducing liability is to control the people who use
the expert system by registering, or characterizing and qualifying
them. Registering the users of expert system is discussed in
Chapter 7. Characterizing and qualifying users involves three steps.
First, developers must assess the degree of computer literacy and
competency in the subject area of the intended users and build the
system accordingly. Second, the users must be made aware of the
level of complexity and scope of the expert system via disclaimers
and documentation. Finally, initial and on-going training is
essential, especially for first-time users of the expert system.
User qualification can be enforced by requiring that the user
specify a registration number for the expert system when requesting
user or technical support. . -
2.2.3.3 Disclaimers
The proper use of disclaimers (Fig. 2.1) will also reduce the risk
of liability for expert systems. Disclaimers should include
information on:
o The characteristics of the intended user of the expert
system;
o The scope and applicability of the system;
o Assumptions made in the development process; and
o Intended uses for the system.
It should be explicitly stated that using the expert system for
purposes other than those intended will produce unpredictable
results, and that neither EPA nor its contractors are responsible
for the consequences resulting from intentional abuse or
unauthorized use of the system..
2-2
-------
OSWER DIRECTIVE 0902B.OOE
., an expert system that
B task is
automates the
to be used for the purposes of:.
5___, and
Recommendations presented by the
system should always be evaluated
by the user for content and applicability
to the problem. Users will be held
responsible for any and all actions based
on the system's recommendations.
Kev
A - name of expert system
B -title for task automated by expert system
C, D, E - activities automated by the expert system
Figure 2.1
Example of an Expert System Disclaimer
-------
OSUER Directive #9028.OOa
2.2.3.4 Liability Scenarios
For now, EPA should consider expert systems to be tools, with the
decision-maker retaining the responsibility for any liability
resulting from the decision. EPA managers are not divested of
liability, but can reduce it by emphasizing the advisory nature of
expert systems. Some researchers suggest that expert systems are
simply automated information retrieval systems and should be held no
more liable than publishers of books. Another point of view
suggests that expert systems be held liable as independent experts,
and thus are fully responsible for all recommendations. In other
words, the expert system is just a tool that people may use at their
discretion, but may in no way hold responsible for the outcome.
Finally, there is the possibility that people may be held liable for
not using an expert system. If a mistake is made that could have
been avoided by using an expert system, the decision-maker might be
found at fault. Under each of these liability scenarios it is
important to think about the extent of liability for each person
involved - - including the expert, the developers, the users, and the
maintenance staff.
2.2.3.5 Private Industry
Developers of EPA systems should consider the liability implications
of EPA-sanctioned software being used by private industry. Expert
systems developed by EPA can have broad ranging effects in the hands
of unauthorized users. Thus EPA should implement a user
qualification system and a. user registration system (see CHAPTER 7)
to mitigate these effects. Additionally, EPA expert systems may
wind up in the hands of the regulated community as a result of a
Freedom of Information Act (FOIA) request.
2.2.3.6 Distribution
Because of the potentially extensive user base of each expert
system, it is necessary to consider the implications of their use on
Regions, states, and industry. Three mechanisms for distributing
expert systems are:
o User registration,
o Mailing lists, and
o Open dissemination.
The extent of risk for misuse of an expert system increases the more
freely it is distributed. The type of distribution is left to the
project manager's judgment based on the sensitivity of the
information "contained in the expert system.
2-3
-------
OSWER Directive *9028.00a
2.3 ORGANIZATION'S CULTURE
The initial approval to proceed with an expert system project and
its final acceptance depend upon an organization's culture. In addition, the
culture will in some way be affected by expert systems technology, if it is
incorporated into the operation of the organization.
2.3.1 Level of Acceptance
The level of acceptance of new technology depends on an
organization's responsibility and risk tolerance. Conservative organizations
that operate in a well-defined environment are often reluctant to try advanced
technical information system solutions. Other organizations view advanced
technology as an opportunity to increase productivity and capabilities, and
thus welcome expert systems.
2.3.2 Developer Initiative
Developer initiative is heavily influenced by an organization's
culture. Organizations that encourage individual initiative will stimulate
more activity in new technologies, and are therefore more likely to develop
grass-roots expert system ideas and projects.
2.3.3 User Acceptance
User acceptance also depends on the nature of the organization. The
successful introduction of new technology is often based on how the users of
that technology are received. In other words, if users are encouraged and
rewarded for their efforts, the hew technology is likely to succeed.
Another factor that affects user acceptance is management
commitment. Positive reinforcement from management encourages user
acceptance. Involving the users in the early stages of the expert system's
development increases their feeling of ownership and thus promotes user
acceptance. If the users are indirectly discouraged or penalized for aborted
attempts, the new technology often remains unused. This is especially
critical during the prototyping stages.
2.3.4 Prototyping
Organizations that embrace new technology should understand the
exploratory nature of rapid prototyping used in developing expert systems.
Rapid prototyping provides the flexibility to make false starts within a
project (see Section 1.7). Because prototypes are not all-or-nothing efforts,
they may be discarded in favor of a new approach as new information is
discovered. The success of expert systems depends on organizations being
tolerant of the exploratory process that might not have immediate benefits.
2-4
-------
OSWER Directive 09028.OOa
2.3.5 Effects of Expert Systems
Changes in the operation of an organization lead to changes within
the organization's culture. Expert systems can cause changes in two ways.
First, in building an expert system to automate a task, the task itself comes
under scrutiny. This in itself often causes changes in the procedure.
Second, expert systems affect the information task itself by changing the
user's input routine, the time involved in finishing the task, the answer and
justification received, and potentially in other quantitative and qualitative
ways.
Two other facets of expert system development that are influenced by
an organization's culture are effects on expert system development staff and
ramifications of early expert system projects. Finally, the organization's
culture determines the amount of emphasis that is placed on early expert
system initiatives. Further efforts in expert systems may hinge upon the
success of earlier ones.
2-5
-------
OSWER Directive #9028.OOa
2.4 CROSS CUTTING CONSIDERATIONS
Several topics of particular importance are addressed in multiple
phases of the system life cycle. These topics include:
o Project Management Flan;
o Project Participation;
o Project Reviews and Quality Assurance;
o Project Approvals;
o Methodologies and Tools;
o Benefit-Cost Analyses; and
o Knowledge Management.
This Section briefly describes each topic from a cross cutting, life
cycle wide perspective. Specific activities relating to each of these topics
are presented in the remaining chapters of this practice paper.
2.4.1 Project Management Plan
The Project Management Flan is the fundamental document for planning
and managing the system life cycle, and is mandatory for every project. It is
first developed in the Initiation Phase and is updated, expanded, and refined
continually throughout the life cycle. Refer to the OSWER Project Management
Practice Paper for additional information.
Several important implications of expert systems for the Project
Management Flan are noted below:
o Explicit determination of the relevance of the application
area and type must be documented in the Project Management
Plan.
o Although prototyping can be an iterative, repetitive
process, justification for each successive prototype must
be documented in the Project Management Plan. The ;
"lessons learned" in each iteration should be spelled out.
o Any modification to the standard life cycle stages and
phases to accommodate prototyping or other aspects of
expert system development must be documented in the
Project Management Plan.
o Knowledge base testing, validation, and maintenance plans
must also be spelled out in the initial plan and refined
over time.
2-6
-------
OSUER Directive *9028.00a
2.4.2 Project Participation
Information management problems and systems to address them require
the participation of organizations and individuals with a diverse set of
experience and skills. These range from an in-depth understanding of
pertinent agency programs to expertise in specific information management
technologies. Successful projects require that certain important roles be
specifically assigned to organizations and individuals.
Expert system development efforts will require the roles common to
all system development projects. These are OSWER Program Management, OSWER
Program Execution, Project Management, Project Execution, Quality Assurance,
and Procurement. Within Project Execution there are the following specialized
roles.
o [Domain Expert] - the expert provides the information or
expertise for the expert system. The expert acts as the
functional expert; providing expertise, defining the
boundaries of the domain, and providing validation for the
expert system. The expert's qualification is to have in-
depth knowledge of a functional area.
o [Knowledge Engineer] - .the knowledge engineer elicits the
information or expertise from the expert. The knowledge
engineer's role in an expert system project is to gather
the information or expertise of 'the expert and to codify
that information using expert system techniques. The
knowledge engineer's qualifications are to be familiar
with expert system development tools and methodologies.
o User - the user is a person who requires the functional
expertise of the expert on a regular basis. This
. individual, over time, will become a functional expert.
The user's role in an expert system project is to use the
system to perform tasks that would normally require input
from an expert. The user's qualifications are to have a
willingness to work with new systems and to understand the
functional area to the degree that he or she can identify
when the expert system is providing incorrect
recommendations.
2.4.3 Project Reviews and Quality Assurance
Independent review of the products of the system life cycle is
performed to ensure that the project team .is proceeding in an appropriate
direction to effectively solve the information management problem. The ,
reviews address programmatic issues, technical considerations, and project
management. The reviews provide feedback to the project team as well as
advice to the individuals required to approve the project. Project reviews
2-7
-------
OSUER Directive #9028.00a
and other quality assurance activities are performed throughout the life
cycle.
Several important aspects of expert system project reviews and
quality assurance are noted below:
o The reviews to be conducted and other planned quality
assurance activities are documented in the Project
Management Plan.
o Quality assurance applies to all aspects of the project,
and quality assurance is a continual part of the project
effort.
o Formal reviews are structured to ensure a level of review
commensurate with the nature and scope of the information
management problem and potential solution.
o The results of reviews are included in each decision paper
developed at the end of each stage of the life cycle
management process.
o Knowledge base testing and validation will require
separate verification of data (facts) and logic (rules)
o Detailed test cases and review and approval of the system
by the expert will be required.
o Problems particular to knowledge bases will have to be
isolated and corrected such as subsumed rules,
contradictory rules, unreachable goal states.
2.4.4 Project Approvals
Formal approvals are provided throughout the system life cycle to
ensure that OSWER management supports the project and is in agreement with the
chosen project direction. These approvals are provided at a level
commensurate with the nature and scope of the system. Review and approval
thresholds established for conventional.systems may not be appropriate for
expert systems. The following is a review structure that considers impact,
risk, level of interest, usage distribution, development effort, and system
type follows:
o Impact can be measured in several different ways. One is
to estimate savings in terms of some combination of
professional labor hours saved, support labor hours saved,
computer time saved, or reduced contractor assistance.
o Risk is a measure of a system's susceptibility to
incorrect or partially incorrect results and the probable
significance of such results. The measurement of risk
2-8
-------
OSUER Directive *9028.00«
takes into account such factors as the cost of incorrect
or partially incorrect results, a cost-benefit analysis of
the system, and a risk analysis of the "regret factor."
o Level of interest in a particular system is a measure of
its visibility. Exposure of the system within the agency,
the government as a whole, or the public eye, will
influence the degree of review and approval required for
the system.
o Usage distribution will affect review and approval
requirements. One possible distribution categorization is
expert systems developed for individual use systems, work
unit distribution, Agency-wide distribution, and
distribution beyond the Agency. Each level of
distribution would require increasing levels of review and
approval.
o Development effort is a measure of the amount of resources
invested in the development and maintenance of an expert
system. Investments should be calculated in terms of
dollars and labor hours (usually full time equivalents)
required to develop the expert system.
o System type or application domain of a given expert system
will also affect its level of review and approval. Some
domains such as hazardous waste control will demand near
perfection and resulting intensive review. Other domains
such automated forms processing may require much less
review.
2.4.5 Methodologies and Tools
Expert systems projects will clearly use different methodologies and
tools than those used in conventional systems development. Several important
expert system considerations in the use of methodologies and tools are noted
below:
o While the LCM Guidance does not mandate the use any
specific methods or automated tools, it does require that
choices be made explicitly. Proper selection of
application types and areas facilitate identifying
required expert system shell characteristics, which in
turn lead to specific products.
o Successive prototypes must be justified individually and
must be documented in the Project Management Plan.
o Methodologies and tools should be considered from a full
system life cycle perspective.
2-9
-------
OSUER Directive #9028.OOa
o The methodologies and tools for each phase or stage are
selected no later than the end of the preceding phase or
stage.
2.4.6 Benefit-Cost Analyses
The important decisions of the system life cycle and the prescribed
management approvals often hinge on two key questions: "How much will it
cost?", and "Are the benefits worth the cost?" Benefit-cost analyses are
particularly important in expert system development because of the unique
hardware, software, and developer skills that may be involved. However,
quantifying these measures for expert systems can be difficult, so alternate
measurement techniques should be considered. These may include:
o Lost opportunity cost analysis,
o Relative worth methods,
o Quantitative bounding, and
o Qualitative bounding.
2.4.7 Knowledge Management
Knowledge management is the process of formally controlling the
initial development of a knowledge base and all subsequent changes or
additions to it. It ensures that the integrity of an expert system is
maintained throughout its life cycle. Knowledge management is closely related
to configuration management. Refer to the OSWER Configuration Management
Practice Paper for a more complete description.
There are several advantages of applied knowledge management over
less structured knowledge base development. As the agency develops multiple
expert systems over a period of time, opportunities will present themselves to
gain leverage from previous efforts. Well-structured knowledge bases will
permit the development of reusable modules of knowledge stored in a "knowledge
library." A reusable "knowledge dictionary" will enhance expert system
development by reducing the time and cost required to specify commonly used
knowledge -base elements.
«
As OSWER increases its expert system development activities, it
should make a deliberate effort to seek out other systems currently under
development in order to identify opportunities to share knowledge. If
multiple applications can access common knowledge bases, they can be cost-
justified more easily and accurately.
Linking modules together to form a knowledge base may impact system
performance adversely or positively. On the one hand, the control structure
required to interface the different modules will consume some portion of the
processing capabilities of the system. On the other, breaking a knowledge
base into modules may permit loading and running only those modules needed for
the current consultation.
2-10
-------
OSUER Directive #9028.OOa
Knowledge management is an ongoing process because expertise is
constantly evolving and changing. Proposed changes and additions to the
knowledge base must undergo formal review before they are made to ensure the
integrity of the system. Some requested changes may be of such significance
that they must be addressed through a separate iteration of the life cycle.
Requests which will have the effect of changing the types of data and/or
knowledge to be processed by the system generally should be handled in this
manner.
2.5 RESOURCE REQUIREMENTS
Resource requirement considerations comprise a significant portion
of the planning process for expert system development. Particular attention
should be paid to scarce resources such as an expert's time. Expected
requirements for resources in each phase of. the life cycle are discussed
below.
2.5.1 Experts
The single most important resource in the development of an expert
system is the time of the domain experts. Their time is also a scarce
resource because of the demands that placed on them by their organization.
The commitment of an expert's time -is a plus in terms of management support,
but it is also a risk in terms of not wasting this business resource on
fruitless efforts.
2.5.1.1 Initiation Phase
The experts are needed as a resource in the Initiation Phase to
assist in determining the scope of the system because they know the
problem area and solution requirements best.
2.5.1.2 Concept Phase
Experts are needed in the Concept Phase to determine the initial
expert system functions and features. In addition, the experts will
develop a preliminary overview of the problem that allows the
developer to build a good proof-of-concept prototype. Involvement
of the experts is extensive in the Concept Phase.
2.5.1.3 Definition and Design Phase
The experts should be prepared for a significant time investment in
the Definition and Design Phase. During this phase the experts must
explain the problem to knowledge engineers and to management. The
experts are also crucial for the development of the demonstration
prototype of the expert system built in this phase.
2-11
-------
OSUER Directive 09028.OOa
2.5.1.4 Development Stage
When determining the amount of time needed from the experts,
remember the experts are going to be needed more in the initial
startup of development where most of the knowledge acquisition takes
place. As the development progresses, the experts will need to be
consulted on a regular basis. The experts will also be needed
during testing, verification and validation.
2.5.1.5 Implementation Stage
During implementation of the system, experts will be needed to a
lesser extent than during the Development Stage. They will be
needed mainly to resolve any problems that arise during the beta
test. It may be useful to find another expert in the field who was
not involved in the development of the system to serve as a beta
tester. The additional expert will be able to test the system more
rigorously than other beta users, and-could assist in a final
validation of the system.
2.5.1.6 Operation Phase
Experts will be needed to assist in modifying the knowledge base in
the Evaluation Stage. They will also be helpful in determining the
degree to which the system continues to address the information
management problem. The size of the system and the volatility of
its problem domain will dictate how much time will be required from
the experts.
Experts may also be helpful in determining what portions (if any) of
the terminated system to retain in the Archive Stage.
2.5.2 Information about the Problem
Information about the problem will be required to verify that the
problem exists and to justify the system development effort to solve it. The
information will be gathered in the Initiation Phase and will be used to
document the existence- of the problem and, when available, describe solutions
to similar problems.
2.5.3 Knowledge Engineers
Knowledge engineers perform three major functions in the expert
system development life cycle. First, they help structure the problem,
perform feasibility studies, and design the expert system. Second, knowledge
engineers extract knowledge about the problem solving process from the
experts. Finally, .they are responsible for knowledge management in the expert
system.
2-12
-------
OSUER Directive #9028.OOa
Because of their important role in the development process,
knowledge engineers are extensively involved in the expert system project from
the Concept Phase on.
2.5.3.1 Concept Phase
Knowledge engineers perform a significant portion of their work in
the Concept Phase with responsibilities including:
o Selecting the knowledge representation and control
structure,
o Developing the Knowledge Management Flan and System Test
Plan,
o Helping determine the functional requirements of the
expert system,
o Assisting in performing a feasibility assessment for the
expert system approach, and
o Building the Proof-of-Concept prototype.
The amount of time spent by the knowledge engineers will vary with
the size and complexity of the application.
2.5.3.2 Definition and Design Phase
The number of knowledge engineers is dependent upon the estimated
size and complexity of the problem. It is recommended that two
knowledge engineers interact with the experts in order to provide
different points of view in the knowledge acquisition process.
During design, the role of the knowledge engineers is to assist in
identifying potential knowledge sources and to help the experts to
explore possible avenues in the problem domain. However, it is
important for the knowledge engineers to help the expert scope the
problem domain down to a reasonable size in order to design a
technically feasible expert system.
2.5.3.3 Development Stage
The knowledge engineers will be needed throughout the Development
Stage. The project schedule hinges on their level of commitment.
Initially, the knowledge engineers will be involved extensively in
documenting the expert's knowledge. The knowledge engineers also
provide support to the programmers; in some cases, the knowledge
engineers will perform both knowledge engineering and programming.
Finally, the knowledge engineers will interact with the experts,
programmers (if used) and management during the testing and
validation phases of development.
2-13
-------
OSUER Directive #9028.OOa
2.5.3.4 Implementation Stage
Knowledge engineers will play a much lesser role during this stage
than the Development Stage. They will be needed mainly for
contacting the experts if any problems arise during beta testing.
They will also be involved in analyzing the comments and suggestions
of the beta users, and making any necessary revisions to the system.
Knowledge engineers may also be involved in the training process and
will provide expertise for the expert system documentation.
2.5.4 Programmers
Programmers are responsible for the development of software
associated with the expert system. This includes placing the knowledge
acquired from the experts into the development environment, writing
interfaces, developing support programs, and integrating the expert system
into the existing information processing architecture. In some projects, the
knowledge engineers may serve as the programmers.
2.5.4.1 Concept Phase
Programmers who have experience with expert systems are necessary to
help determine the testing methodology in the Concept Phase. They
also assist the knowledge engineers build the Proof-of-Concept
prototype.
2.5.4.2 Definition and Design Phase
It is the responsibility of the programmers to incorporate the data
extracted during knowledge acquisition into the specified
development environment. They can be used to assist in the
requirements definition process and as a technical resource for
programming issues that need to be addressed in the design. They
are also useful for building the Demonstration Prototype during this
phase.
2.5.4.3 Development Stage
In the Development Stage, the programmers are responsible for
helping codify the experts' knowledge and for writing support
programs. They will be the key people in building the knowledge
base, developing the interfaces, and responding to the users
suggestions on how to improve the expert system. Their services
will continue through testing and validation where they will be
responsible for any changes necessary before the Implementation
Stage.
2-14
-------
OSWER Directive #9028.OOa
2.5.4.4 Implementation Stage
The programmers will be involved to a lesser extent in this stage
than in the Development Stage. They will be involved in making any
necessary revisions to the system following the beta test, and may
provide some input for the documentation on the system. Programmers
could also be involved in technical support to the beta users, if
there are any problems at the program level or problems with
hardware and software compatibility.
2.5.5 Data Collection
Effective data collection facilitates the development of expert
systems. Most of the data collection is performed in the early stages of the
life cycle.
2.5.5.1 Concent Phase
Initial data collection functions are limited to those necessary in
determining the conceptual model's features and inputs, user
training, system benefits and costs, end users, and system scope.
Potential sources of information include other expert systems that
are already built, in the Initiation Phase, or under development.
Resources are mostly those of the knowledge engineer's time.
2.5.5.2 Definition and Design Stage
Further data collection required during the Definition and Design
Phase involves information about the problem domain, the development
environment, and the targeted delivery environment. To properly
define and scope the problem domain, possible data sources should be
identified and realistic objectives should be established.
In the selection of the development and delivery environments, much
information is required in order to make a knowledgeable decision.
For example, in order to assess the cost of the system, the hardware
requirements should be determined for both development and delivery
environments.
2.5.5.3 Development Stage
In the Development Stage, data will be collected on user feedback,
management feedback, and the development process. Also, any data in
the form of spreadsheet and database will need to.be documented in
. the development plan. Use the data management plan as a guide in
the data collection process.
2-15
-------
OSUER Directive #9028.OOa
2.5.6 Licensing Fees
It is important to consider licensing fees for both the development
and the delivery environments when estimating the resource requirements for
the expert systems. The licensing fees constrain the number of copies of
software that may be used in developing the expert system, and also the number
of copies of the expert system that can be distributed.
2.5.6.1 Concept Phase
Planning for software runtime licensing fees is necessary in this
phase because both the costs to build and to field the expert system
are based on the number and location of the users.
2.5.6.2 Implementation Stage
As the system is nearing distribution to field users, a staff member
will be needed to check all licensing and run-time fees with the
company who distributes the shell. All fees must be paid, contracts
and agreements must be activated, and any other legal matters must
be addressed at this time.
2.5.7 Testing
The testing methodology needs to be set in the Concept Phase. Both
conventional software and expert system testing methodologies should be
considered. Testing is done by the development staff if performed during the
Development Stage. The testing includes validation of the data, knowledge
base, and logic flow.
During the Implementation Stage, the system will be tested in the
field by potential end users. Comments received from those testing the
product will be incorporated into the final revisions. This testing will help
to ensure user acceptance of the system.
2.5.8 Maintenance
Maintenance plans should be started in the Concept Phase. Questions
such as to who will be responsible for maintenance, when it will be performed,
how volatile and dynamic the problem domain is, and what role the users will
play need to be addressed. A staff member or contractor will be needed to
continue to maintain the expert system, as well as provide technical support
to the beta users during the Implementation Stage.
Maintenance is necessary in the Operation Phase to ensure that any
previously undetected errors can be corrected. Maintenance also provides a
means to take advantage of hardware upgrades, new releases of system software,
arid applications packages that enhance the system's performance.
2-16
-------
OSUER Directive #9028.OOa
2.5.9 Hardware
The development hardware will be needed from the Concept Phase
through the Development Stage. For more information on selecting suitable
hardware refer to Section 5.4.1.3. The beta users will provide their own
hardware during the Implementation Stage.
Changes to the expert system, prompted by advances in hardware and
software, may be necessary during the Operation Phase. Changing the hardware
platform of an existing system should be avoided if at all possible, but if
the existing platform is no longer supported or becomes obsolete, it may be
unavoidable. Such a change will likely warrant a separate iteration of the
system life cycle.
2.5.10 Software
By the end of the Concept Phase an. appropriate development vehicle,
including the expert system shell, environment, and language, has been
selected. The key issues here are how to encode the knowledge and how to
integrate the software packages to expedite development. Topics to consider
are:
o Interfaces between any external and/or utility programs.
Also, any links between quantitative techniques and the
expert system should be addressed.
o . Identify and use existing software development aides such
as intelligent editors, debuggers and source code
managers.
Copies of the run-time version of the system will be made during the
Implementation Stage and sent to the beta users. Over the span of the
Production Stage, the knowledge base will likely be modified one or more
times. These modifications may entail interfacing the expert system with
newly developed systems or upgrading the system's shell. All software and
data that are unique and essential to the system's operation should be
archived for possible future use.
2-17
-------
OSUER Directive 09028.OOa
CHAPTER 3
INITIATION PHASE
3.0 INTRODUCTION
Initiation is the first of five major phases in the OSWER system
life cycle. The other phases are Concept, Definition and Design, Development
and Implementation, and'Operation. This phase provides the first formal
description of the information management problem and secures the resources
needed to examine the problem and potential solutions. The most significant
activities of the Initiation Phase include:
o Confirming the existence of a problem, providing additional guidance
where appropriate, and approving (or rejecting) the commitment of
resources to begin the Concept Phase. This activity is performed by
program management.
o Preparing an Initiation Decision Paper that identifies and describes
the information management problem and brings it to the attention of
OSWER program management. This activity is performed by one or more
program organizations.
o Preparing a Project Management Plan to describe the initial approach
for managing and conducting this phase and future actions under the
remainder of the life cycle. At this point, the Project Management
Plan provides detail 'for the Concept Phase; it is revised and
enhanced in later phases.
Clearly identifying and describing the information management
problem is critical to the successful development of an appropriate solution.
How the problem is defined during Initiation will shape the.analyses and
decisions of the subsequent life cycle phases.
During the Initiation Phase, there'should be no
assumption that the solution will necessarily
be an expert system!
Even in cases where the system developers have preconceived notions
that an expert system is the preferred solution, the analysis in the
Initiation Phase should not be intentionally slanted to use expert systems.
Instead, the primary goal of this phase is to describe the information
management problem objectively, independent of potential technological
solutions.
3-1
-------
System Us Cycto
for
Export Systems
OSWER DIRECTIVE #3028 05s
Anpfarantrtfan
Objectives
Decisions
New
Products
To describe the problem In clear, technology-Independent
terms upon which organizations can agree
To determine whether staff or other resources will be
devoted to defining and evaluating alternative ways
to respond to the Identified problem In the Concept Phase
Project approach decisions Including:
who will manage the project
who will participate In development
- what reviews are necessary (and by whom)
- what approvals are necessary (and by whom)
Execution decisions Including:
what Is the problem
- what organizations are- involved
what are the sources of data, the scope of the
solution, and the Information requirements
such as report generation
Initiation Decision Paper
Project Management Plan
Figure 3.1
Initiation Phase
Objectives, Decisions, and Products
-------
OSUER Directive «9028.00a
3.1 OBJECTIVES
Figure 3.1 graphically illustrates the objectives associated with
the Initiation Phase.
The primary objective of the Initiation Phase is to describe the
problem in clear, technology-independent terms upon which all pertinent
organizations can agree. If an expert system may be the technology chosen to
resolve the problem, then the project manager and software developer should
consider the expert system application success factors listed in Section 3.3
before completing this phase.
The second objective for the Initiation Phase is to determine
whether staff or other resources will be devoted to defining and evaluating
alternative ways to respond to the identified problem in the Concept Phase.
Committing resources beyond the Concept Phase is premature at this point. The
use of rapid prototyping as a tool to work through these issues is a feasible
approach at this stage and in the Concept Phase.
3.2 DECISIONS
There are three types of decisions made in the Initiation Phase:
project approach, execution, and continuation. . Figure 3.1 graphically
illustrates these decisions.
3.2.1 Project Approach
The project approach decisions address the organization of the
project and the participants in the project activities such as system
acceptance testing, reviews, and approvals.
Decisions are focused on project management and controls,
establishing: .
o Who will manage the project,
o Who will participate in development,
o What reviews are necessary (and by whom), and
o What approvals are necessary (and by whom).
3.2.2 Execution
The execution decisions address the scope and specific features of
the system. These decisions address programmatic, technical, and system
support related issues.
Programmatic issues include deciding what the problem is, whether an
organization is currently responsible for solving the problem, and which
organization(s) should receive the proposed automated solution. In resolving
3-2
-------
OSUER Directive #9028.OOa
these issues the emphasis in the Initiation Phase should not be on a special
technology (e.g., optical disk technology, expert systems, etc.) but should be
technology-independent.
3.23 Continuation
The continuation decision confirms that a defined information
management problem exists and is significant enough to warrant further
investigation. The decision confirms that the information management problem
is beyond the capabilities of existing systems and that developing a new
system has merit. The decision made here also has to do with resources, and
this is an assessment of return on investment, and that is that the investment
of people and dollars will produce results which are desired by the
organization in terms of productivity.
Technical issues include determining sources of data, the scope of
the solution, and the information requirements such as report generation.
When there is a mix of conventional systems' and expert systems -- called
hybrids or embedded systems, new information needs may be associated with the
conventional systems being considered; in this case, the Initiation Decision
Paper should identify the new information needs. A systems support issue is
to determine who will be responsible for the system once it is fully
operational.
3.3 SUCCESS FACTORS
Several factors that can impede success if not considered in the
Initiation Phase are described below. This Section focuses on perceptions of
what expert systems can and cannot do, and it offers solutions or
alternatives.
3,3.1 System Development Bias
Because expert systems are a relatively new technology, individuals
and groups involved in initiating a new system may have little or no
familiarity with them. They may even have a negative impression of their
capabilities. On the other hand, some parties may tend to find an expert
system solution to every problem, whether it is appropriate or not. Before
undertaking an expert system project, it is important to determine that the
problem has not already been solved by other types of conventional programming
such as modeling, decision support, databases, or linear programming.
3.3.2 Scoping the Problem
Expert system development projects must be reasonably scoped from
initiation onward. Failure to do so can lead to developing a solution to the
wrong problem or tackling overly complex or nebulous problems.
3-3
-------
OSWER Directive #9028.OOa
3.3.3 The Problem is Well Suited to ES Technology
The key characteristic of an application that will help the project
manager or software developer determine whether expert system technology
should be used is the type of problem. If the problem is purely algorithmic
or procedural in nature, then it can be addressed by conventional technologies
more efficiently than by expert systems. If the type of problem requires
symbolic reasoning (as described in CHAPTER 1, Expert Systems Defined, then
the problem may be suitable for expert systems technology.
3.3.4 An Expert is Available
Expert systems are developed by taking the specific knowledge of an
established expert and putting it into a system. Some systems may incorporate
the knowledge of more than one expert, while others reflect the knowledge and
strategies of a single individual. Finding the right expert(s) is a key step
in building an expert system. If no true expert exists, then the problem may
be too nebulous and ill-defined to be effectively addressed by an expert
system.
3.3.5 The Problem Domain has Bounds
Once an expert is identified, the project manager or software
developer should consider whether the set of solutions to the problem has
boundaries or whether there are infinite solutions. The problem cannot have
an infinite set of solutions. The solutions that will be considered by the
system must be determined in advance. When the expert system attempts to work
with information from near the periphery of the problem domain, it may yield
unpredictable results. The problem domain must be readily defined using
empirical knowledge such as facts, rules, or algorithms. It should not
require common sense (i.e., knowledge everyone takes for granted, but few can
articulate) or sensory data (i.e., vision or sense of smell).
3.3.6 An Expert Can Solve the Problem in Less than a Week
The project manager or software developer should also consider the
complexity of the tasks that are to be automated with an expert system. The
tasks should neither.be too difficult nor too trivial for a human expert. A
task requiring more than a week to solve without computer support is almost
certainly too large to be thoroughly delineated using rules. However, if the
problem can be parsed into subproblems, it may be manageable. On the other
hand, a task requiring only a few moments to solve might be automated more
efficiently using conventional technologies.
3.3.7 A Significant Return on Investment is Anticipated
Since expert system development requires a significant investment in
terms of people and money, the expected return on that investment must be well
understood, along with the means of measuring the return. Can the relative
3-4
-------
OSUER Directive «9028.00a
success of the expert system's performance be assessed? The Initiation Phase
should address this issue before moving to the Concept Phase.
3.4 PRODUCTS
Many products are produced and/or updated in the course of the
system life cycle. The Initiation Phase products are described below.
3.4.1 Initiation Decision Paper
The Initiation Decision Paper describes the information management
problem and justifies undertaking the next phase of the life cycle. Whether
an expert system is being considered as the solution or not, the Initiation
Decision Paper should be written in technology - independent language and
should focus primarily on the scope and magnitude of the problem at hand. It
may highlight the attributes of the problem that are suggestive of a possible
expert system solution (refer to Section 3.3, "Success Factors"). The key
parts of the Initiation Decision Paper that will affect expert system projects
are:
o Mission areas addressed: boundaries and domain for the
problem must be clearly established.
o Description of the problem: conventional ways of
describing information management problems such as flow
charts and data flow diagrams may be only partially useful
for an expert system-type problem. Different decision
[representation] techniques include decision trees, rules,
or just plain English explanations may be preferable.
o Overall project approach: methodologies and tools may be
different for expert systems. These issues are discussed
as cross-cutting considerations in Chapter 9.
Prototyping may be used as a tool to develop the analytic basis for
the decision paper.
3.4.2 Project Management Plan
Developing the preliminary Project Management Plan, the second
product associated with the Initiation Phase, is an important step because it
results in the collection of the following information:
o Preliminary life cycle cost estimate,
o - Detailed estimate for the Concept Phase,
3-5
-------
OSUER Directive 09028.OOa
o Results of threshold analysis of appropriate levels of
review and approval,
o Methodologies and selection of tools to be used in the
Concept Phase, including prototyping,
o Results of benefit/cost benefit analyses,
o A list of experts who will provide the knowledge base of
the expert system, and
o A list of users who will ultimately be responsible for the
success or failure of the system.
Techniques for justifying expert systems are in many ways alike to
those used to justify other information management systems. Some of the
techniques used to justify expert systems are described below.
o Benefit Analysis - This approach takes advantage of
tangible and quantifiable expert system benefits and
attempts to quantify the qualitative benefits. Depending
upon the measures of success of the expert system defined
early in the development cycle, there may be significant
economical advantages to building an expert system.
Managers need to be aware of the applicability of this
approach to justifying expert system projects and the
measurability of benefits such as "improved productivity"
and "improved accuracy" such as what mistakes have cost in
the past. Also, how would the resources be applied if the
expert system is not deployed?
o Cost Savings Analysis - A second approach to justifying an
expert systems involves the cost savings. While expert
system development may seem expensive, the cost of
remaining with the current system over time may .prove to
be more expensive.
Depending on the magnitude and type of problem being considered,
several expert system methodologies and tools may be used in later phases.
These will be identified in the Concept and Definition and Design Phases.
Expert system-specific issues include life cycle adjustments including
consolidation of phases and stages and use of prototyping techniques, and
special considerations for testing, validation, and maintenance of knowledge
bases. Refer to the OSWER Project Management Plan Practice Paper for more
information.
3-6
-------
OSUER Directive *9028.00a
3.5 ACTIVITIES
Activities that result in the products listed in Section 3.4 must be
completed by the project management and development team. Involving the users
and gaining management commitment are two additional activities that must be
completed.
3.5.1 Involving the Users
»
The Initiation Phase requires a great deal of participation of
individuals not specifically involved in developing the expert system. If
domain experts are not the ultimate users of the system, the actual users must
also be represented in describing the problem.
3.5.2 Gaining Management Commitment
One of the goals of the Initiation Phase is to decide whether to
commit resources to solving the perceived problem. This will entail
generating the necessary management support. Failure to acquire management
support now cannot.be compensated for in later phases. Therefore, high level
management from all involved organizations should participate in system
initiation.
3-7
-------
OSWER Directive #9028.OOa
CHAPTER 4
CONCEPT PHASE
4.0 INTRODUCTION
The Concept Phase is the second of five major phases in the OSWER
system life cycle. This phase of the expert system life cycle provides a
high-level description of the solution to the problem and describes the
functional, knowledge, and data requirements of the task and evaluates
alternative solutions. A prototyping exercise may be used to develop the
concept, but it is not an end in itself. The Concept Decision Paper is
separate from any prototyping activity.
All aspects of the problem solution are considered in the Concept
Phase, and many can be addressed by attempting to develop a prototype.
Following are the questions which are raised at this time.
o. Is an expert system the best solution for the problem?
o * What type of feasibility assessment is necessary?
o What knowledge is necessary to solve the problem, and
where and how it can be obtained?
o How should the knowledge be managed?
o What features and functionality are expected of the expert
system?
o What combination of knowledge representation and control
structure best suits the problem?
6 How should the project team be assembled?
o What qualifications must the system developers and
reviewers meet?
o How can the use of expert systems to solve the problem be
justified?
Because several key decisions are made in this phase, it is
important to review the objectives of the Concept Phase that are described
below.
4-1
-------
OSWER DIRECTIVE #9028.00e
rihAfcnano*
Deafen
Objectives
Decisions
Determine the feasibility of an expert system solution
to the problem
Identify cost-effective system solution to the problem
Approach decisions Include:
- Is the problem important and are expert systems a viable
solution
- what Is the system life cycle strategy
-what are the methodologies beat suited to the project
- what Is the procurement plan
- when and from whom will funding be obtained
Execution decisions Include:
- what are the high level functional and data requirements
- what are the knowledge management Issues
- what Is the knowledge representation and control structure
of choice
- who la to be on the development team and what Is their role
- who are. the uaera .
- can the problem be aolved with existing systems
- what is the overall architecture
will the system Interface with existing systems
- what will be the delivery environment
- how will technical, programmatic, and other risks be
addreased
what will be the maintenance strategy
Continuation deciaiona Include:
does the Information management problem continue to exist
- does the concept allow the development team to proceed
to the next phase
- are sufficient funding and other resources available
[slew * System Concept
Products * Concept Decision Paper
System Test Document
Acceptance Teat Document
Knowledge Management Plan
Data Management Plan
Requirements Definition
Figure 4.1
Concept Phase
Objectives, Decisions, and Products
-------
OSUER Directive #9028.OOa
4.1 OBJECTIVES
The objectives for the Concept Phase (Fig. 4.1) include a problem
definition, requirements, feasibility, and approach. All of the objectives
evaluate or describe an approach to solve the information processing problem.
The first objective of the Concept Phase is to confirm the existence of the
information processing or knowledge-intensive problem.
The second objective is to identify high level requirements for a
solution to the problem. These requirements should focus on the nature of the
problem and the user's needs, and not directly address expert system issues.
The third objective is to determine the feasibility of an expert
system solution to the problem. This requires a study of both the
applicability of expert systems to the project and the capabilities of other
information technologies. The final objective of the Concept Phase is to
identify a feasible, cost-effective expert system approach. In order to meet
these objectives, several decisions must be'made concerning the approach to,
execution of, and continuation from the Concept Phase.
4'. 2 DECISIONS
Decisions within the Concept Phase (Fig. 4.1) focus on selecting
development methodologies, assembling resources, and evaluating the expert
system approach to solving the problem.
4.2.1 Approach
There are five decisions to be made in the approach to the Concept
Phase. The approach decisions select development methodologies and determine
the procurement plan.
4.2.1.1 Approach Evaluation
First, is an expert systems a viable solution? This emphasizes the
need to study alternative 'approaches.
4.2.1.2 Life Cycle Strategy
Second, what is the system life cycle strategy? This decision
includes the dependence of the project on rapid prototyping.
4.2.1.3 Deve1opment Me thodolo gie s
Third, what knowledge acquisition, Development, testing, and
maintenance methodologies are best-suited for this project? It is
important to be aware of the effects of each methodology on the
development process, and choose methodologies best suited to the
4-2
-------
OSUER Directive #9028.OOa
application. To some degree those choices may emerge from the
prototyping approach.
4.2.1.4 Procurement Plan
The fourth decision is determining the procurement plan. How are
funds to be obtained and allocated for developing the expert system?
Does the project manager need to acquire any hardware or software?
4.2.1.5 Funding Determination
Finally, what portion of the development cycle should be funded at
this point and by whom? Is funding available for the entire
project?
4.2.2 Execution
There are several decisions to be made in the execution portion of
the Concept Phase, most of which focus on the high level requirements of the
expert system and resource assembly.
4.2.2.1 Functional'Requirements Definition
The first decision during execution is determining the high-level
functional requirements of the system. .This decision focuses on the
nature of the problem and the users' needs rather than on specific
expert system issues.
4.2.2.2 Knowledge Management Issues
The second decision involves identifying the knowledge management
issues for the project. Are there other expert systems in the same
problem solving area that might provide insight? How should the
knowledge for this system be acquired, placed into a knowledge base,
and maintained?
4.2.2.3 Knowledge Representation and Control Structures
Third, what is the knowledge representation and control structure of
choice? If there are no restrictions on the choice of a development
environment or expert system shell, which of the knowledge
representations is best suited for this particular problem? Given
the knowledge representation, which control structure provides the
best inferencing? Refer to CHAPTER 5 for explanations of control
structures.
A.2.2.4 Data Requirements
Fourth, what --if any -- are the data requirements and design
parameters for this system? Is the expert system supposed to access
4-3
-------
OSUER Directive #9028. OOa
information from external data bases or programs? If so, what is
the data's format, where does it reside, and how can it be accessed?
Some expert systems acquire all data by asking the user for
information.
4.2.2.5 PropfE Te>am Assembly
The fifth decision focuses on assembling the project team. Who is
on the development team and what is their role? It is important to
know both the qualifications and the availability of potential team
members .
4.2.2.6 User Needs Determination
The next decision involves identifying the users, their needs, and
their level of sophistication. This is important for properly
conducting the feasibility assessment and determining the functional
requirements for the expert system.
4.2.2.7 Technology Selection
The seventh decision looks at the need for an information processing
system in general. Can the problem be solved with existing systems?
If so, then it may be more effective to modify 'existing systems than
to build a new one. If an expert system is chosen, then which shell
or .language is appropriate?
4.2.2.8 Architecture Planning
The eighth decision entails planning the overall architecture of the
expert system, including the structure of the knowledge and the
data.
4.2.2.9 Integration Issues
To what degree the expert system will interface with or be
integrated into existing information-processing systems.
4.2.2.10 Delivery Environment Determination
What elements of the system will be centralized and what elements
will be distributed? This is important in selecting a development
environment, determining licensing fees, and so on.
4.2.2.11 Organizational Issues
How will technical , programmatic, and other risks be addressed?
Determining the procedures for handling risks also affects the
development process and the focus of the expert system.
4-4
-------
OSUER Directive #9028.OOa
4.2.2.12 Maintenance Strategy
The final execution decision is selecting the maintenance strategy.
Who will maintain the system? How will users' comments be
incorporated? How often will the system be updated? . Answers
arrived at here may be changed later. Refer to CHAPTER 7 for more
details.
4.2.3 Continuation
There are three decisions in the Concept Phase that are a
continuation of decisions made during the Initiation Phase. The continuation
decisions require an objective evaluation of the nature of the problem, the
proposed solution, and the resources available to the project.
4.2.3.1 Problem Continuation
First, does the information/knowledge management problem continue to
exist? If yes, proceed with the project as it is currently defined.
If not, consider refocusing or discontinuing the project.
4.2.3.2 Solution Adequacy
Second, does the expert system concept address the problem
sufficiently to permit continuing to the Design and Definition
Phase? If it does, then continue to the next phase. If it- does
not, then look for alternative expert system solutions -- including
hybrid expert system/conventional system applications -- or consider
other information processing technologies. Prototyping may be used
to seek answers to these questions.
4.2.3.3 Project Funding
Finally, are sufficient funding and other resources available for
the system life cycle? If resources are available, proceed to the
next phase. If not, look for additional resources or postpone the
project until they become available.
These decisions are documented in several new products. The
products of the Concept Phase are listed below.
*
4.3 PRODUCTS
There are seven products in the Concept Phase (Fig. 4.1):
o System Concept
o Concept Decision Paper
o System Test Document
o Acceptance Test Document
4-5
-------
OSUER Directive «9028.00a
o Knowledge Management Plan
o Data Management Plan
o Requirements Definition.
A.3.1 System Concept
The System Concept is the key document of the Concept Phase. It
describes the results of the functional analyses and both the data and
knowledge requirements for the expert system.
4.3.2 Concept Decision Paper
The Concept Decision Paper should explain the benefits of selecting
the expert system approach that were determined in the Concept Phase. This
document should be clear and comprehensive so that OSWER managers can make an
informed decision whether to approve a project.
4.3.3 System Test Document
The System Test Document presents information on the testing to be
performed by the development team. It provides a chronology of the system
testing process, including strategy, plan, data and knowledge, methods,
procedures, results, and recommended actions.
4.3.4 Acceptance Test Document
The Acceptance Test Document presents information on testing to be
performed by OSWER program personnel. At the end of the Concept Phase, the
Acceptance Test Document contains only the testing strategy.
4.3.5 Knowledge Management Plan
The Knowledge Management Plan describes the approach to acquiring,
utilizing, maintaining, and reusing knowledge throughout the project. In the
Definition and Design Phase it will include the development team's choice of
knowledge acquisition methodologies, knowledge representation, control
structure, and maintenance strategy.
*
4.3.6 Data Management Plan
The Data Management Plan reflects the project's data management
approach. This document supplements the Knowledge Management Plan by
describing access to external data in databases, application programs, and
historical files.
4-6
-------
OSUER Directive #9028.OOa
4.3.7 Requirements Definition
Information from the experts and users are compiled to form the
guidelines used in the Definition and Design Phase of the expert system
development life cycle. Information to be included in -the Requirements
Definition specific to expert systems are:
o Target level and focus of output
o [Explanation facilities]
o User interface
o External interfaces
o Target performance (speed and accuracy) of the system.
4.4 SUCCESS FACTORS
Several factors that affect an expert system's success should be
considered in the Concept Phase. They fall under the general topic areas of
organizational and resource issues, the target users of the system, functional
requirements, and knowledge representation and control structure. Several
success factors are identified and general advice is given on applying them.
The first major success factor is specifically identifying and
effectively implementing the products of the Concept Phase. Proper
utilization of the Concept Phase-leads to advantages such as assembling an
efficient team, good Requirements Definition and resource estimation, and
clear management direction.
Another high-level success factor in the Concept Phase involves
implementing expert systems around well-conceived ideas. These can be
developed through prototyping, but this is not a substitute for the c'oncept.
Expert systems should be implemented to solve problems that are not
effectively handled using conventional technologies. This success factor can
be ensured by carefully studying the problem and looking for conventional
alternatives.
Success factors relating to specific areas within the Concept Phase
are described below.
4.4.1 Organizational Issues
There are several organizational issues that affect the outcome of
an expert system project. Organizational issues include scheduling
flexibility, management commitment, available hardware and software
availability, and implications assessment.
4.4.1.1 Scheduling Flexibility
The first organizational issue is that the concept of an expert
system is not critical to the solution. This implies that there is
4-7
-------
OSUER Directive #9028.OOa
freedom for the project to change directions and use a solution
other than an expert system. System flexibility is important for
expert system projects because they frequently need major
modifications as ideas transform and new directions are discovered,
either through prototyping or conventional analyses.
4.4.1.2 Management Commitment
Management commitment to an expert system project is often cited as
one of the most common reasons for project success. Management
commitment comes in many forms: resources -- including dollars,
people, and equipment; continued involvement and supervision; and
support in times of conflicting objectives. Management needs to be
aware of the benefits, limitations, and differences between expert
systems and conventional information processing tasks at the
initiation of the project (see CHAPTER 3). A constant flow of
information and updates is required to keep management involved,
interested, and committed to the project.
4.4.1.3 Impact Assessment
Another organizational issue is that the ramifications of the expert
system --on people, the'task, or the organization itself -- are
thought out in advance. Take time to review the implications of
changing current processes caused by adding an expert system.
4.4/2 Resource Issues
Major resource issues stem from the proper evaluation and resource
estimation of expert system projects. While this is a difficult task, proper
evaluation of the project goes a long way to ensure that the resources are
best allocated to an expert system, and would not generate higher payoffs on
other projects or other technologies.
Resource estimation is a critical factor influencing the success-of
expert system projects. It is important that the developer accurately
estimate the time, staff, and financial resources required to complete the
project. Another resource estimation factor is allowing for experimentation
or exploration through prototyping, both often necessary in the development of
an expert system. In order to benefit from these factors, the project team
should evaluate other expert system projects and set aside extra funds for
necessary exploratory work.
4.4.3 Measures of Success
Measures of success for the expert system project are clearly stated
and agreed upon. . Measures of success can be quantitative -- increased
productivity, or time savings --or they can be qualitative -- such as
improved morale. These measures need to be identified and defined prior to
4-8
-------
OSUER Directive #9028.OOa '
the start of the project so that they are used to guide and evaluate the
expert system project.
4.4.4 Target User Level of the System
Success factors associated with the target user level of the system
involve an understanding of the users' needs and a specification of the
content and level of complexity of expert system's outputs. Some effort
should be put into determining how the expert system's recommendations will be
used.
4.4.4.1 User Identification
The first factor is identifying the intended users of the expert
system. This leads to a clear idea of both the focus of the output
and its level of sophistication. Users at an entry level position
will require a different focus -- one that pertains directly to
their task --as well as a specific degree of sophistication. The
output should be very specific and in terms that they can
understand. Advanced users, on the other hand, are often better
served by succinct answers that they can use as guidance.
Identifying the intended users is also essential when they vary in
their degree of computer literacy. Special help features may be
necessary depending on the level of computer experience of the
target users.
Another success factor in determining the target output of the
system is the degree of training that is necessary. If the expert
system is targeted entirely toward training, then it should focus on
providing as much information to the user as possible. Training
systems often try to diagnose what problems a user is having with a
concept, and then set up exercises to correct the problem. Expert
systems that are 'intended to have only incidental training benefits
1 focus on solving the problem with a minimum amount of overhead and
their output tends to be more succinct.
4.4.4.2 Content of System Outputs
A second facet of this issue is understanding how the output of the
system is to be used. Once the users are identified and the target
level of the output is set, how are the users to treat the expert
system's recommendations? Are the recommendations to play the role
of a checklist, an assistant, or an expert? It is critical that the
users' of an expert system understand the degree to which they can
rely upon the output of the expert system and the level of their own
responsibility.
4-9
-------
OSUER Directive #9028.OOa
A.U.4.3 Level of System Outputs
The final success factor in the target output involves specifying
the underlying knowledge that will be incorporated into the system.
Given the application, what information is necessary to make an
informed decision to accurately solve the problem? It is important
to gather this information from the expert before beginning the next
phase.
4.4.5 Functional Requirements
It is important that functional requirements are carefully defined.
It is quite difficult to fulfill moving specifications under any circumstances
and especially given time and resource constraints. The requirements should
be written, agreed upon, and modified only when the time and resources are
also changed commensurately.
Justifications for the expert system's recommendations need to be
clear and specific. Explanation facilities -- such as the why and how queries
often found in expert systems - often consist of replaying the logic used by
the system to arrive at a conclusion. While this is sufficient for some
applications, others require more in-depth justification including causal
relationships and assumptions made.
4.4.6 Knowledge Representation and Control Structure
When an expert system shell is already selected before the Concept
Phase is completed, the choice of a knowledge representation and a control
structure is somewhat limited. This poses a challenge to the developers when
the nature of the problem does not fit well with specific knowledge
representation scheme supported. The feasibility study should show whether or
not alternate development environments would provide a cost-effective means to
successfully avoid this problem.
The proper knowledge representation and control structures expose
the natural constraints of a problem and allow the expert system to be built
efficiently and easily. At times, such as when the knowledge representation
is predetermined by the available expert system shell, there may be no
alternative but to proceed. When there is a choice, the nature of the problem
should be used to select a suitable knowledge representation and control
structure.
4.5 ACTIVITIES
There are several introductory activities to be undertaken during
the Concept Phase. These involve three types of participants in an expert
system project - the users, the expert, and the project management. These
activities will lay the groundwork for the major activities of this phase.
4-10
-------
OSWER DIRECTIVE #9028.003
Knowledae Representations
RULE
IF the LIST has not been upgraded
AND age is greater than 10 years
THEN upgrade must be performed
FRAME
[Storage
AKO
I
Tank :
AKO: Storage
Has put: Liner
Type: imdergroud
IS-A
fank-1:
SA: tank
ondition: good
Type:
Type:
Material: steel
Contents: oil
OBJECTS
(Tank-1 XTank-2) (Tank-3) (Tank-4
Messages: create, destroy, upgrade
OAV Triplets
object
tank
tank
tank
attribute
type
type
...3Q9
value
steel
fiberglass
10
Hybrid-Rule-OAV
IF NOT Tank:condition:upgrade
AND Age > 10 (parameter)
AND Tank:type:steel
THEN upgrade UST
Figure 4.2
-------
OSUER Directive #9028.OOa
4.5.1 Identifying the Users
Identifying the users of the application seems apparent, but in fact
can be quite difficult. [End user] identification is critical to proper
design and development because, several expert system features are determined
by the intended end users, including: .:
o The delivery environment and geographic distribution,
o Target level of the expert system's recommendations,
o Type and sophistication of the user interface,
o Level of explanation in the justification mechanism, and
o The focus of the knowledge base itself.
The ultimate users will work with the expert system on a regular
basis. Other people who will be indirectly involved with the use of the
expert system -- such as managers, support technicians, and maintenance staff
-- are not considered users. A precise definition of the users leads to a
more focused and productive expert system. .
After all of the people who will be directly involved in the
operation of the expert system are identified, a profile of the group should
be developed. This profile contains information on the specific task that the
users need the expert system to perform, the level of job training and
experience of the users, and their range of computer literacy. The conceptual
model of the expert system can then use this user profile to specify features
that focus on the specific needs of the intended users.
4.5.2 Selecting the Expert
Selection of the expert may seem like an obvious task, but should be
given adequate consideration. Several factors should be considered when
selecting an expert, or determining if an appropriate expert exists. This
generally leads to an expert who is cooperative and easy to contact and
schedule appointments for [knowledge engineering] sessions. The knowledge
engineer should also elicit support and commitment from the expert's superior.
The expert needs to have the requisite qualities to facilitate
knowledge engineering. The expert should be methodical, consistent, and
articulate in dealing with the knowledge engineer. With this type of expert,
it is easy to get the expert more interested in the technology.
The expert must not be intimidated by the thought of an automated
system doing their job. Explain to the expert that the system is not meant to
be a replacement, but to aid the expert in making more informed decisions or
to help people in the field where the expert might not be available. The
expert should understand what is required. It is the responsibility of the
knowledge engineer to properly inform the expert what is expected.
4-11
-------
OSUER Directive *9028.00a
4.5.3 Communicating with Management
Management support is crucial to the success of any information-
processing project and especially for expert systems. Many people are
uncertain of the capabilities and practicality of expert systems. The way to
obtain and maintain management support is to provide good channels of
communication in the Concept Phase and throughout the life cycle.
In the Initiation Phase, management needs to be informed of the
capabilities of expert systems: their benefits, limitations, and feasibility.
Expert systems should be presented to management in terms of improving
productivity in current work or in the capability to accomplish tasks that
were previously not feasible. Introduction to expert systems is often best
accomplished via written material followed by a briefing. This allows the
managers to learn about expert systems at their own pace, with a minimum of
time commitment. Next, management needs to understand the specific expert
system application area. This will improve the quality of decisions made
throughout the life -cycle of the expert system.
Throughout the development process management needs to be kept
informed on the status and needs of the project. It is best to use written
memos and reports to keep an audit trail of the development process (see
Section 7°.6.9). Another useful form of communication is the demonstration.
An expert system application already in use can be used to demonstrate the
general features of an expert system. Demonstrations of working prototypes
should be given for management throughout the development cycle to illustrate
the progress and the direction of the project.
4.5.4 Management Commitment and Understanding of Prototyping
Management commitment is the key to the success of an expert system
project. There must be a perceived value in excess of perceived cost and risk
in the project to retain management commitment. Managers must also be aware
of the different techniques used in developing expert systems. Techniques
used in expert system development that may be unfamiliar to managers include
knowledge acquisition, prototyping, and knowledge base verification.
4.5.4.1 Initial Management Commitment
The need for management commitment is obvious, but is often
overlooked. Initially, management commitment involves approval and
initiation of the expert system project. Management commits the
resources to begin the project, including funding for hardware,
software, and human resources.
4-12
-------
OSUER Directive #9028.OOe
4.5.4.2 Continued Management Commitment
Continued management support as development proceeds is sometimes
more tenuous, and in some ways more important. Once the project has
begun, there is a tendency to reallocate people on the expert system
project: when conflicts arise. An important form of management
commitment is supporting the expert system project even when budget
cuts or other resource constraints are imposed. Once the project
has started management needs to be objective when comparing the
expert system project to other information-processing projects, and
not be adversely influenced by the mystique of expert systems.
4.5.4.3 Prototyping Issues
One of the major benefits of prototyping techniques is their
inherent suitability to expert systems. Expert system prototypes
can be developed, evaluated, and modified relatively quickly and
inexpensively. This allows additional opportunities to be explored
with minimum risk. Managers need to be aware of the ramifications
of prototyping (see Section 1.7) and be prepared to utilize
prototypes to their fullest.
4.5.4.4 Life Cycle Issues
Because of the significant differences between the development
processes of traditional information processing projects and expert
systems -- such as the disposable prototype, management needs to be
alerted to expert system life cycle issues. Major issues include
knowledge management, knowledge acquisition, prototyping, and
knowledge base validation. Proper understanding of these issues
(see CHAPTER 1) will allow managers to reap the full benefits of
expert systems.
4.6 KNOWLEDGE MANAGEMENT
"In the knowledge lies the power" is a quote by Randall Davis of the
Massachusetts Institute of Technology, who has helped develop several famous
expert systems such as MYCIN. In this statement he emphasizes the importance
of the knowledge placed- in an expert system. Knowledge is more than
information, just as information is more than data. Knowledge results from
the capability to use information. A planned methodology is necessary to
properly gather, use, and maintain knowledge for expert systems. Knowledge
management comprises six parts, described below.
4.6.1 Knowledge Acquisition
In the knowledge acquisition process, we identify, locate, and
gather the information and mental processes used^o solve problems. The first
step in knowledge acquisition is to determine what knowledge is necessary to
4-13
-------
OSUER Directive #9028.OOa
solve Che problem, including any background information. The required
knowledge acquisition information includes the:
o Knowledge necessary to solve the problem;
o Location of the knowledge -- including experts, texts,
historical data, and test data;
o Selected knowledge acquisition methodology; and
o Plans for storing and manipulating the knowledge.
4.6.2 Knowledge Representation
Knowledge representation is important for four reasons. First, it
simplifies the task of accessing the acquired knowledge in the system.
Second, a good knowledge representation scheme exposes the natural constraints
of the problem, which makes it easier to solve. Third, the knowledge
representation is the media in which the knowledge is stored, updated, and
modified. Thus a good knowledge representation will expedite maintenance.
Finally, selecting the proper knowledge representation will facilitate
knowledge transfer by. making the knowledge transparent and easy to manipulate.
The information needed for the Knowledge Management Plan from this part is the
preliminary choice of knowledge representations for the expert system.
4.6.3 Knowledge Base Creation
Knowledge base creation consists of taking the acquired knowledge,
placing it in a knowledge dictionary, transforming it into the selected
knowledge representation, and placing the acquired knowledge into the
knowledge base.
4.6.3.1 Knowledge Dictionary
The knowledge dictionary for expert systems is similar to the data
dictionary for databases. The purpose for the knowledge dictionary
is two-fold. First, it documents the structure of the knowledge
base contents. This facilitates both testing and maintenance.
Second, the knowledge dictionary enforces consistency in naming and
transforming knowledge. This is especially important when several
programmers are working simultaneously on the knowledge base.
4.6.3.2 Knowledge Transformation
The transformation process is not always straight-forward because of
the inherent restrictions and constraints of any knowledge
representation. In addition, there are concessions that have to be
made for operational purposes. These concessions generally consist
of additions to the knowledge base that are solely for the purpose
of controlling the flow of operation, calling external data sources,
4-14
-------
OSWER Directive #9028.OOa
or updating user interfaces. These additions are necessary for the
operation of the expert system, but can cause confusion in later
attempts to utilize the information in the knowledge base.
Assumptions are also placed into the knowledge base based the way
j; the control structure will operate. These assumptions add to the
J problems of maintaining or reusing the knowledge, and should be
minimized.
4.6.3.3 Knowledge Modularization
It is particularly important to properly modularize the knowledge
base. Knowledge about a problem domain can usually be decomposed
into smaller components. For example, in a diagnostic expert system
for a piece of equipment, the problem might decompose into
diagnosing the electrical system, the mechanical system, and the
structural portion of the equipment. Each piece of the problem can
be placed in a separate module within the knowledge base.
Modules within knowledge bases serve several purposes. First, they
allow common information to be stored in one centralized area.
Second, they allow the developer to think about the sub-problems
independently and thus simplify the development process. Third,
modules facilitate the verification process by isolating errors.
Fourth, maintenance is simplified because updates and corrections
are made to independent and easily identified parts of the knowledge
base. Finally, software reuse is promoted by separating knowledge.
4.6.4 Knowledge Base Validation
Validation of the knowledge base is necessary to ensure proper
performance of the expert system. A knowledge base validation plan helps
developers perform a comprehensive review of the expert system and its
capabilities. Information on when the validation is to occur, the type of
verification to be performed, and where the test data is to come from should
be included in the plan.
It is important to realize that value can be derived from any expert
system project, even those that are not implemented. The knowledge acquired
in building an expert system is at times more valuable than the expert system
itself. This is due to the fact that the problem-solving process is now
documented, and can be reviewed and stored.
4.6.5 Knowledge Base Maintenance
The need for maintenance arises from changing or evolving user
requirements, the need to correct errors, revisions in regulations or
procedures, and advances in the state-of-the-art. The maintenance plan must
recognize these sources of change and be prepared to react accordingly. Once
the changes are made, there should be a knowledge maintenance plan and
4-15
-------
OSUER Directive 49028.OOa
configuration management plan for testing and validating the knowledge base to
ensure its continued usability.
4.6.6 Knowledge Base Reusability
The ability to reuse knowledge bases saves resources. Knowledge
management should stress the reuse of knowledge to the extent that it is a
viable alternative. A major portion of the creation of a comprehensive
knowledge base is the inclusion of background information and underpinning
knowledge. Once this information is stored in the knowledge base, the
knowledge to solve the specific problem is easy to enter. Because of the
relatively high ratio of background knowledge to application-specific
knowledge, there is an evolving process of reusing knowledge for expert
systems performing in the same problem area.
Knowledge reusability is still a research issue at this time.
Several large projects are underway to develop general-purpose knowledge
bases, but practical applications are limited in number. There are some
aspects of knowledge reusability that might be helpful to OSWER project
managers:
o . Identifying other expert system projects within the same
problem-solving area,
o Obtaining these systems and their documentation,.
o Conducting an extensive litesature search,
o Building a-well-structured, modular knowledge base,
o Maintaining the knowledge base, and
o Using simple versus compound knowledge representations.
4.6.6.1 Other Potential Expert Systems
The developer should be aware of other potential expert systems
within the same problem-solving area. If others have been
developed, try to obtain a copy of the knowledge base or the
knowledge acquisition sessions. 'If other expert systems are
planned, the knowledge -engineer needs to obtain as much general
knowledge as possible in the sessions, document assumptions and
paths not taken, and store the results of the knowledge acquisition
in an easily accessible manner.
4.6.6.2 Modular Development
The second step in practical knowledge reusability is to build a
well-structured knowledge base. This implies making the knowledge
base modular and as complete as possible. Assumptions should be
4-16
-------
OSWER Directive #9028.OOs
recorded, and the entire knowledge base well documented both on-line
and in hard copy. Inclusion of procedural pieces and operational
code should be kept to a minimum. Finally, the knowledge base
should strive for clarity rather than efficiency or elegance.
4.6.6.3 Maintenance Procedures
The final step of knowledge reusability is in the maintenance of the
knowledge base. Maintenance should adhere to the same principles as
the development, or its reusability will decrease over time.
4.7 FEASIBILITY ASSESSMENT
A feasibility assessment is essential to understand the scope of an
expert system project and the costs and risks associated with it. This
feasibility study parallels that of the LCt\ Guidance, A feasibility
assessment should be conducted by the knowledge engineer(s) in conjunction
with the expert(s) and the intended users. The issues involved in the
feasibility assessment process are technical in nature. The objectives of
this process are to determine the suitability and to define the functional
requirements of the proposed expert system.
4.7.1 Determining the Viability of an Expert System Solution
The first step in determining the feasibility of an expert system
project is to determine the need for an expert system. After the problem
itself is verified as being important, the approach to solving it is studied.
If other solutions exist, it is important to consider whether expert systems
would add sufficient benefits to justify their use. Another consideration in
this step is the possibility of combining expert systems with other computing
techniques. Problems involving forecasting, optimization, or voluminous
information management might be best solved using a hybrid approach.
4.7.2 Technical Issues
Technical issues to consider in the feasibility assessment include
the nature of the problem, characteristics of knowledge management,
availability of expertise, software interfaces and integration, and
verification and validation.
4.8 KNOWLEDGE REPRESENTATION
Knowledge representation is recognized as a crucial part of any
artificial intelligence project. Choosing the correct knowledge
representation facilitates the development and maintenance of an expert
system. Figure 4.2 provides a graphic representation of the following aspects
of knowledge representation.
4-17
-------
OSWER Directive «9028.00a
4.8.1 Rules
Rules are the most common form of knowledge representation for
expert systems, following a simple if-then format. Rules are used to
represent the rules-of-thumb or [heuristic] used in solving problems. People
find it easy to describe complex processes in simple steps using if-then
rules. Rules are good for capturing procedural knowledge. Because they
capture only one small piece of the problem per rule, rules are not applicable
to all problems.
4.8.2 Frames
Frames resemble database records in that they store chunks of
information together in fields. They differ from records, however, by adding
class-subclass structure, including more information to individual fields, and
by incorporating procedural inferencing mechanisms into the information.
Frames are used to store knowledge that has an important structural
component (e.g., hierarchies of chemical information.) Frames are often used
to store case histories, because they store relevant information in the same
place which facilitates retrieval.
4.8.3 Objects
Objects are similar to frames, except that they normally rely on
message-passing for communications and are more independent. Objects tend to
be used in expert systems in a limited form, as scaled down versions of
frames.
4.8.4 Object-Attribute-Value Triplets
Object-attribute-value (OAV) triplets and parameters are a simple
form of knowledge representation used for storing rudimentary information.
They are generally used in combination with other knowledge representations
such as rules.
OAV triplets store information in three parts. The object portion
contains the name of the primary concept that is being represented by the
triplet (e.g., an apple). The attribute portion contains information on what
particular aspect of the object we are referring to (e.g., the color). The
value stores the actual information describing that particular attribute of
the object (e.g., red).
4.8.5 Parameters
Parameters are similar to the variables of various programming
languages in that they are typed and contain values. Different types of
parameters include numbers, strings, sets, lists, and records. The difference
is that parameters as a knowledge representation also contain information
about their use that is utilized by the expert system such as an explanation
4-18
-------
OSWER Directive #9028.OOa
of what the parameter represents, questions to ask the user to find the value
for a parameter, and information on where the parameter is used. Parameters
store basic knowledge that is neither procedural nor structured.
4.8.6 Hybrids
Hybrid knowledge representations are becoming more prevalent because
they make up for the deficiencies of individual knowledge representations.
Hybrid knowledge representations typically consist of a combination of rules
and frames, or rules and objects. This combines a structural storage
representation frames or objects with action-oriented representation rules to
capture complex knowledge more easily across a broad range of problem domains.
4.9 CONTROL STRUCTURES
More than one control structure may be applied to most knowledge
representations. The selection of a control structure depends on the. nature
of the problem, the knowledge representation chosen, and the desired
performance of the system.
4.9.1 Forward Chaining
[Forward chaining] applies to the rule knowledge representation. It
takes data and information, applies the appropriate rules from the knowledge
base, and presents recommendations. Because of this, forward chaining is also
known as data-driven and event-driven-processing. Forward chaining is best
for problems that have a specific set of inputs and a large number of possible
outcomes including design and configuration applications.
4.9.2 Backward Chaining
Backward chaining also applies to rules, and works backward from the
conclusions, until it finds a combination of rules and data that support a
given solution. Backward chaining works best on problems that have a limited
number of possible outcomes and several inputs including diagnostic and
classification applications. Backward chaining results in intelligent
questioning, because only information pertaining to the current hypothesis is
requested.
4.9.3 Inferenc ing
Inferencing is a general term that refers to several other control
structures, including pattern matching, [backtracking], constraint
propagation, and search techniques. ("These terms are defined in the glossary
in APPENDIX B.) Variations on these techniques are used in expert systems
that use knowledge representations other .than rules, such as frames and
objects.
4-19
-------
OSWER Directive #9028.OOa
4.9.4 Hybrids
Hybrid control structures attempt to reap the benefits and overcome
the limitations of one or more individual techniques. Common hybrid control
techniques combine forward and backward chaining, or chaining and pattern
matching. The hybrids are typically applied to large problems and
applications using hybrid knowledge representations.
4.10 THE SYSTEM CONCEPT
The conceptual model contains comprehensive information concerning
the details necessary to properly design the expert system. Information
contained in the conceptual model includes;
o Specific problem area and solution type,
o Knowledge representation,
o Control structure,
o External data sources and interfaces,
o User interface,
o Target output level, and
o Justification mechanism.
The conceptual model should also be the basis for making a
preliminary- assessment of the expert system shell or development environment
subject to the Definition and Design Phase in Chapter 5.
4.11 PROOF-OF-CONCEPT PROTOTYPE
Prototyping is an iterative process of design and redesign during *
development. Each prototype builds on the successful completion of the
previous prototype. Prototyping involves several stages, the first of which
begins in the Concept Phase. The Proof-of-Concept Prototype is initiated
during this phase.
The Proof-of-Concept Prototype is a very small working model of the
expert system developed to assess preliminary feasibility of the problem
domain. It is developed by following a narrow line of reasoning for a specific
topic to its conclusion. If using rules as a knowledge representation, the
number of rules should range from 5 to 25. This prototype shows inherent
strengths and weaknesses of further development. It will answer the question
of whether or not another technology or approach is feasible. It also allows
the knowledge engineer to assess the framework, scope, interfaces and issues
borne out during knowledge acquisition and the design process.
4.12 ASSIGNING A. PROJECT TEAM
The project team consists of everyone directly involved in the
development of the expert system. The aspects of project staffing addressed
here are the size and composition of the project team. The size and
4-20
-------
OSWER Directive #9028.OOa
composition of the project team will be directly related to the nature of the
problem and the conceptual model developed earlier in this phase. The number
of people in project management will remain approximately the same for expert
system development.
4.12.1 Expert Involvement
Experts can vary in number from one to 10 or 50. Fewer experts are
needed on small projects that have adequate development time. In the case of
specialized projects, only a few experts may be available. Multiple experts
are used on large projects, and particularly those combining expertise in
several related areas. Multiple experts might also be used in parallel on
projects that have limited development time. It is important to not use all
of the available experts in developing an expert system. Some should remain
external to the project until it is time to verify the expert system's
knowledge base.
4.12.2 Knowledge Engineer Involvement
Knowledge engineers are key members of the expert system development
team. There should be one senior knowledge engineer and one or more junior
knowledge engineers depending on the chosen knowledge acquisition methodology
and on the number of experts. Some knowledge acquisition methodologies such
as the two-on-one interviewing technique require multiple knowledge engineers.
Except for extremely small expert system projects, there should be multiple
knowledge engineers to increase accuracy and completeness of the knowledge
acquisition and transformation process. In projects where multiple experts
are involved at the same time, a guideline is to have two knowledge engineers
per expert.
4.12.3 Proerammer Involvement
The number of programmers on the development team depends on the
role of the knowledge engineer and the amount of conventional programming that
needs to be performed. If the knowledge engineer Is capable of and available
for programming the expert system, then programmers per se may not be
necessary. Programmers are important for integrating.the expert system into
existing operations and for adding external functions to the expert system
especially hybrid expert/conventional systems. Estimating the number of
programmers is otherwise the same as with conventional systems.
4.12.4 Reviewer Involvement
The number of reviewers is generally higher for expert systems
projects due to the need for external experts for knowledge base verification.
4-21
-------
OSUER Directive #9028.OOa
CHAPTER 5
DEFINITION AND DESIGN PHASE
5.0 INTRODUCTION
After an appropriate problem domain has been selected, the
Definition and Design Phase begins. This phase is critical because it lays
the framework for the development activity to follow.
5.1 OBJECTIVES
The following objectives of the Definition and Design Phase (Fig.
5.1) must be met in order to assure a smooth Development'Stage.
o Problem definition - the definition of the problem domain
has to be refined and reaffirmed.
o Development environment selection - the success of the
Development Stage hinges on careful selection of the
development environment. The goal is to select an
environment that is a good match for the problem domain.
Refer to Section 5.5 for more details.
o Delivery environment selection - the consideration of the
delivery environment is also an important objective of
this phase. The requirements of the end users should be
incorporated in the early stages of system design.
o Evaluation of expert system shells applicable to the
problem domain.
o System design - ultimately, an overall design of the
system should be achieved.
«
5.2 DECISIONS
Several decisions must be made in the Definition and Design Phase
(Fig. 5.1). The main decision involves the choice of an appropriate
development environment. The importance of a good selection at this stage
cannot be stressed enough. Refer to Section 5.5 for guidelines in the
"election of a proper development environment. Also, the targeted delivery
environment must be carefully selected to provide a smooth transition from
development to implementation. In addition, teams should be assembled to
design and develop the system.
5-1
-------
OSWER DIRECTIVE #902B.OOa
BtflnKlon
lS»»lgn
Objectives
Decisions
New
Products
Refine the problem definition
Select a development environment that Is compatible with
the problem domain
Select the delivery environment
Achieve a satisfactory overall system design
Decisions to be addressed Include:
selection of an appropriate development environment
selection of a targeted delivery environment
- Identification and assembly of design team members
Identification and assembly of development taam members
One-page Design of the System
Detailed Design of the System
Development Plan
. Test and Validation Plan
Support Plan
Development Environment Evaluation
Figure 5.1
Definition and Design Phase
Objectives, Decisions, and Products
-------
OSUER Directive #9028.OOa
5.3 PRODUCTS
There are six products developed during the Definition and Design
Phase. These include design documents, management plans, and evaluation.
5.3.1 One Page Design Document
The One-Page Design Document is a brief description of the expert
system. It gives the expert systems scope, purpose, and a preliminary
assessment of the knowledge representation and control strategy which
applicable to the problem domain. This document is described fully in 5.7.1.
5.3.2 Detailed Design Document
The Detailed Design Document is an expansion of the One Page Design
Document. Further refinement is made of the items in the one page design
document. Refer to Section 5.7.2 for a further discussion of this document.
5.3.3 Development Plan
This plan is derived from the detailed design document. It outlines
the resources needed for the Development Stage. Also, schedules are firmed up
in this plan.
5.3.4 Test and Validation Plan
This plan is developed to specify how testing and validation should
be handled. It will contain a methodology for designing tests for the expert
systems. It will also list resources available during testing and specify
when testing and validation will take place. Key people in the validation
process are named in the plan.
5.3.5 Support Plan
; This plan contains information on what support is available. All
resources and the time they are needed are specified in this plan. Refer to
Section 2.5 for information on what resources are necessary in the Definition
and Design Stage.
5.3.6 Development Environment Evaluation
This document outlines potential development environments. It lists
strengths and weaknesses of each. The factors that need to be addressed in
this document are primarily from the detailed design document. Refer to
Section 5.5 for procedures for selecting a development environment.
5-2
-------
OSUER Directive #9028.OOa
5.4 SUCCESS FACTORS
During the Definition and Design Phase, both the development and
delivery environments must be considered as integral components of the system
design. Some problems fit neatly into commercial shells, while others do not.
The various development environments offer a variety of mechanisms for
knowledge representation, control structure, and conflict resolution. These
must all be evaluated with care to avoid problems later during the Development
Stage. Common expert system design success factors are listed below.
5.4.1 Validate the Development Environment
The desired development environment features selected in Section
4.11 are validated during this phase. Various software, hardware, and
technical issues should be addressed in the selection of a specific product.
5.4.1.1 Hardware and Software Availability
The first consideration in selecting a development environment is
determining the hardware and software currently available to the
user base. This impacts, and may highly constrain, the choice of
both development and delivery environments.
5.4.1.2 Software Issues
Explore the wide variety shells first to try to find a match with
the problem domain. Shells can provide a quick solution to many
problems. Refer to Appendix A for expert system shell evaluation
factors. Do not attempt to use complex languages without proper
training, appreciation for the difficulty of the task, sufficient
time for development, or the availability of experienced
programmers.
If a shell has been selected as a development environment, be sure
that the tool is affordable, easy to install, and easy to use. An
overly expensive tool dilutes the overall benefits derived. Choose
a tool that can accommodate the problem. Furthermore, the tool must
be able to handle the necessary data. A superior shell can be
modified to provide additional features.
5.4.1.3 Hardware and Technical Issues
Special development hardware may be required to run .the tool. The
additional cost must be considered in the selection process. The
development environment should be able to scale up well.
Integration with desired data sources, programs, and output
interfaces should be readily accommodated. Establish reasonable
goals for satisfactory performance in terms of speed and memory use.
Migration to a suitable delivery vehicle should be easy and
inexpensive.
5-3
-------
OSUER Directive #9028.OOa
5.4.2 Verification and Validation
Verification and' validation methodologies must be determined at this
stage. Verification techniques are the methods used to determine that the
expert system has been built correctly. Verification techniques for the
software, knowledge base, and interfaces should be derived from the expert -
system's functional requirements. Steps taken and methods used to verify the
expert system need to be mapped from the components up through interfaces and
software module interactions. The operational points of the expert system
that need to be validated -- such as scope, effectiveness, and compatibility -
- are identified at this point.
Validation techniques are the methods used to determine that the
expert system conforms to the functional requirements and can be used as
intended. The validation techniques should also be identified in the
Definition and Design Phase. Issues such as the need for external experts,
and types and location of test cases should* be thought out.
5.4.3 Delivery Environment
When establishing a design for the delivery environment, there are
several things to be aware of. These include:
.o The environment's flexibility, availability, and
consistency with equipment currently being used,
o Licensing fees- required for distribution of the system.
Be sure ask the vendor about licensing fees when selecting
a shell.
o Graphics requirements for the Production System.
o Problems caused by excessive data'migration from the
mainframe to a PC. If it appears that there will be
excessive data migration, perhaps it is a case when a
mainframe implementation should be used.
o The migration from the development environment to the
targeted delivery environment may be technically
impossible. Careful selection of development and delivery
environment will avoid this problem.
5.4.4 Rationale and Justification Features
Several decisions should be made during this phase regarding the
necessary rationale and justification features of the development environment.
The s e include:
5-4
-------
OSUER Directive #9028.OOa
o Specifying the extent to which help is available in the
software and in which areas,
o Clarifying what constitutes an effective rationale for
posing questions and for issuing recommendations, and
o Determining a consistent method for ranking
recommendations.
5.4.5 User Interface Issues
The end user interface is the most important portion of the expert
system. User involvement in selecting the desired features should be .included
as early as possible to ensure the success of the system. Some issues to
address include:
o Incorporating interesting, user-friendly screens and
- system features,
o Providing convenient data input for the end user without
the advanced editing functions needed only in development,
o Ensuring adequate system response time-, especially for
calculation-intensive applications, and
o Creating query interface capabilities that support
flexibility and sophistication.
5.5 SELECTING A DEVELOPMENT ENVIRONMENT
*
In the Concept Phase, a preliminary assessment of the development
environment was made. During the Definition and Design Phase, the actual
selection of the development environment is made. Several factors are
involved in selecting a development environment,, including the:
o . Choice of delivery environment (hardware, software, and
degree of integration with existing systems),
o Type of problem including whether or not it is a hybrid
expert system/conventional system problem,
o Suitable knowledge representations, and control structures,
and
o Necessary interfaces.
5-5
-------
OSUER Directive 49028.OOa
g, j?. 1 Features
Some of the features of a development environment are outlined to
give a better understanding of how to select one.
5.5.1.1 Declarative and Procedural Elements
Knowledge bases can be composed of declarative and/or procedural
elements. A declaration is a statement of fact or an assertion that
some data item has a given value. An example of a declarative
statement is: "The average annual temperature at weather station X
is 66 degrees." In contrast, facts may also be derived by
procedural methods. The above fact could have been represented by a
procedure to compute the average annual temperature from raw data.
A knowledge base consisting solely of declarative statements is very
explicit. Such a knowledge representation is readily updated by
users with minimal training. However, procedural additions can
greatly increase the power and flexibility of the knowledge base.
Procedures handle abstract data and can allow for changes in
information without altering the knowledge base.
An effective tool incorporates both declarative and procedural
knowledge representations. Development environments that offer
frames often support both types of representations. Frames are
groups of slots- that can contain declarative statements of facts
and/or procedural attachments .to determine facts. (See Section
4.8.2. for more information on frames).
5.5.1.2 Grouping and Modularity
Grouping and modularity are two related techniques that allow for
faster development and greater extendibility of knowledge bases. If
ah expert system application is very complex, the problem domain can
be separated into modular knowledge bases and the resulting segments
grouped into specialized areas. In this manner, each knowledge base
or group of knowledge bases can focus on a narrow aspect of the
overall problem. To expand the system, the developer can alter a
particular subset of the problem domain, while maintaining the
integrity of the other knowledge bases.
Validation of an expert system is a much simpler task if each
component, knowledge base is a manageable size and can be tested
independently. The chosen development environment should include
the ability to separate a large problem domain into smaller distinct
knowledge bases. Furthermore, it is necessary to have some means of
communication between the segments, such as parameter passing or a
[blackboarding] mechanism.
5-6
-------
OSUER Directive #9028.OOa
5.5.1.3 Documentation Needs
Clear documentation of the knowledge base is extremely important
during the development of an expert system. Some tools represent
the knowledge base in a very readable, English-like format that is
almost self documenting. The developer mus: be able to review the
knowledge quickly and easily at all stages of development. Also,
concise documentation is necessary for validation and maintenance,
especially if these tasks are to be performed by someone other than
the developer.
5.5.1.4 User Issues
Another factor involved in the selection of a development
environment is its ease of use. This is the result of the user
interface, and is determined by the type of interface provided by
the development environment and the effort of the developers.
5.5.1.5 Developer Issues
There is often a trade-off between the sophistication or power of an
[expert system development environment] and its ease of use. In
other words, expert system computing environments that are best-
suited for difficult problems are often more difficult for the
developer to use. It is important to take into consideration the
complexity of the problem and the skill of the developers when
selecting a development environment. Trade-offs between development
and run-time environments should be made explicit.
5.5.1.6 Knowledge Management Issues
The developer needs to be able to describe the knowledge represented
in the system in order to interact with the expert. A careful
consideration of what has been included in the knowledge base is
essential before any expansion takes place. The current knowledge
should be outlined and discussed to expose weaknesses or gaps in
reasoning. The knowledge gaps can then be addressed in future
development. A tool that cleanly breaks out the knowledge into a
hierarchy or decision tree will be of great help in this regard.
Also, some tools incorporate data element dictionaries.
5.5.1.7 Rationale
Most development tools allow the user to ask the system for its
rationale for asking a particular question during a consultation.
The user need only enter "Why?" in response to a system prompt. The
system will generally show its current line of reasoning and explain
why it needs the information in question. The user may choose to
answer the original prompt or perhaps respond with "Unknown" if no
5-7
-------
OSWER Directive #9028.OOa
information is available. The ability of a system to allow a series
of "Why?" inquires to trace the line of reasoning has great value.
5.5.1.8 Justification
A common feature of development tools is the ability to explain the
justification for the recommendations of a consultation. The user
can ask "How?" after a consultation to get a detailed trace of the
line of reasoning used to arrive at the given recommendation. This
trace can be in both text and graphic format, and often allows the
user to change one or more responses to experiment with a "What if?"
scenario.
5.5.1.9 On-line Help
Many expert system shells offer on-line help both during development
and during a consultation. For example, the user can ask the system
~ for syntactic information while developing the knowledge base. This
feature is especially useful for the novice user who needs to learn
a tool quickly. On-line help can also guide an experienced
knowledge engineer through the more complicated tools. Furthermore,
the end user may require on-line help in order to properly respond
to questions during a consultation.
5.5.1.10 Explanation Facilities
A common feature in many development tools involves some type of
conclusion explanation facilities. During a consultation, this
feature is used to describe the line of reasoning that the system
followed in reaching a conclusion. The developer may also wish to
use a consultation tracing feature to aid in debugging the knowledge
base.
5.5.2 User Requirements
When selecting a development environment, it is important to take
into consideration what the user requirements are. Remember that user
acceptance will ultimately make or break your application.
5.5.2.1 Developer Requirements
The level of the developer's computer skills will be a determining
. factor in the selection of a tool. Some tools are designed for the
novice user while others offer complex features that an advanced
programmer will utilize. For example, a beginner's tool is usually
restricted to a specific control structure with a simple editor to
create the knowledge base. However, an advanced user may wish to
experiment with various inferencing techniques and knowledge
representations. Generally, the more flexible tools have steeper
learning curves. For this reason it is recommended that a new
5-8
-------
OSUER Directive «9028.00a
developer start with a more structured shell to gain experience with
expert system technology rather than becoming entangled in the
details of a complicated development environment.
5.5.2.2 End User Requirements !;
^MBA_^^MM«M_^_l ^^^*^_^^_V*«a_MM_ |,,
The end user's requirements should also be considered in tool
selection. How well does the user understand the problem domain?
How computer literate is the end user? The answers to these
questions will determine what level of on-line help and reasoning
explanation the final system should offer. Furthermore, the
delivery system should be more [robust] if the target user is less
experienced. If incorrect or incomplete data is given during a
consultation, the system may need to inform a novice of the
inconsistencies.
5.5.3 Migration to Delivery Environments
The target delivery environment should be considered early in the
system design. First, the available hardware in the user base needs to be
determined. It may be necessary for the end users to buy new hardware or to
upgrade their current systems. A more cost effective solution could be to
choose a shell that is compatible with the user's system. Many tools are
designed solely for the PC environment and do not allow portability to a
mainframe. Other tools are set up to run only on mainframes or dedicated AI
environments. If the expert system is to interact with an existing system,
special interfaces may be necessary. Some shells offer a link-up capability
with databases and external files with data-passing functions. The more-
flexible tools allow the user to customize the shell with specialized routines
that integrate with other languages.
5.5.4 User Interfaces
Several different types of user interfaces must be taken into
consideration during the Definition and Design Phase.-
5.5.4.1 Developer Interface
There are a wide variety of developer interface features available
from different tools. The ease-of-use of a development environment
is a function of the type of features employed in the system. Below
are some of the features that can be found in the developer's
interface:
o Specialized editors that guide the user through the
creation of a knowledge base,
o Decision tree tracing - a graphic or textual listing of
questions and responses, and
5-9
-------
OSUER Directive *9028.00a
o Debugging capabilities with a split screen showing the
inferencing during a consultation.
5.5.4.2 End User Interface
End user interface features also vary from tool to tool. The
features required will depend on the target user as described in
Section 4.4.4. Some of the desirable end user interface features
include:
o Help features - the "Why" feature can be used during a
consultation to have the system explain why the current
question is being asked, and the "How" feature can be used
after a conclusion has been reached to show the line of
reasoning,
o Active images - these can display values and scenarios or
allow the user interact p-ictorially with the system during
a consultation, and
o Re-run consultations - these will save the responses in a
consultation and allow the user to change a set of
parameters to see the outcome in a "What if" situation.
5.5.5 Vendor Information
Information supplied by vendors plays a significant role in the
design and eventual development of the expert system. Given below are several
areas which must be considered.
5.5.5.1. Stability
An initial consideration in tool selection is the stability of the
vendor. A new, small company may have an interesting product, but
may not have as an impressive a track record as a more established
firm. Keep in mind that all post-purchase customer support is
inherently dependent on the vendor's longevity. The following
questions should be posed to assess the vendor's stability:
o How long has the vendor been in business?
o Is the vendor in good financial standing?
o How many systems have been installed?
o What types of organizations have purchased the tool?
o How have other organizations used the tool?
o Are there user's groups for the tool?
5-10
-------
OSUER Directive #9028.OOa
o Are other users satisfied with the product?
o What is it about this tool that makes it worthwhile to
invest in it? and.
^
o What are the risks of developing the application with this
tool as compared to all others?
5.5.5.2 Training
Training issues are especially important in the choice of a vendor
and shell. Expert system development tools range from easy-to-use
yet limited shells to complex, powerful environments. If the latter
is best suited for the problem domain, proper training will be an
essential contribution to the success of the project. The danger
lies in the flexibility of the more powerful tools. Such
environments are geared toward research and allow the developer to
select from a variety of control structures. Training on these
systems is necessary to prevent the user from implementing the wrong
knowledge representation or inferencing strategy. Vendors offer
varying levels of training, with the most extensive generally
reserved for the complex and expensive tools. The types of training
available include:
o Training manuals - should be easy to comprehend and geared
for the user's level of expertise
o On-line tutorials - can be very useful in exploring the
basic features of the tool
t
o Training courses - offer a deeper understanding of the
system's implementation and can be tailored to the user's
problem domain
o Consultants - may be needed to assist the developer with
complex applications yet are available only through the
larger, more established vendors. '
5.5.5.3 Cost
Cost is also a deciding factor when choosing a vendor. Often, in
addition to the price of the initial development tool, there are
other potential costs which should be considered. Listed below are
common items associated with expert system development environments
whose costs must be considered.
o Initial development system,
5-11
-------
OSWER Directive #9028.OOa
o Runtime copies for the end users, or royalties for use on
a network,
o Additional AI language interfaces that may be needed to
run the development package,
o Specialized AI hardware (e.g. LISP workstations) that is
necessary for some of the more powerful tools,
o Update/upgrade for new versions; often the initial cost
can be applied toward that of the upgraded system,
o User support provided by the vendor's trouble-shooting
hotline, and
o Training, as shown in Section 5.5.5.2 in order of
increasing expense.
5.5.6 Documentation
The documentation of the development environment should be both
comprehensive and easy to understand. The more powerful development tools
generally have extensive documentation. Many tools offer tutorials that guide
the developer through example prototypes. A few environments have on-line
documentation. «
5.5.7 Overall Evaluation
An overall evaluation of the development environment should include
all of the items discussed above. Special consideration should be given to
the cost of the tool and associated hardware versus the tool's functionality.
In comparing various environments, try to rank the features in order of
importance for the given application. Be sure to have a well defined problem
domain, formalized during the Concept Phase, before attempting to select a
development environment. ;
5.6 DEVELOPER QUALIFICATIONS
In Section 4.12, the task of selecting a project team was initiated.
The aspects of project staffing that were emphasized were the size and
composition of the project team. This Section will focus on the technical
qualifications of the team members.
5.6.1 Knowledge Engineer ,
The knowledge engineer is responsible for helping to design, build,
and validate the expert system. As such, the knowledge engineer should have
the following qualifications:
5-12
-------
OSUER Directive 19028.OOa
o Good communications skills, particularly in listening,
interviewing, and writing,
o Basic computer skills, knowledge of requirements analysis
and design techniques,
o Fundamental knowledge of expert systems and artificial
intelligence in areas such as knowledge representation and
user interfaces, and
o Some expertise in the problem domain.
Desirable qualities also include the ability to work well with
people, experience in expert system development, a minimal background in
psychology, and experience in systems analysis.
5.6.2 Programmer
The programmer should have fundamental programming skills and
experience in the selected expert system development environment. Desirable
qualities include familiarity with artificial intelligence techniques, system
integration, testing, and validation.
5.6.3 Reviewer/Approver
The reviewer/approver should be familiar with expert systems
technology and LCM issues, cost justification techniques, and other forms-of
information-processing projects.
5.6.4 Expert
The expert should be a recognized leader in the field in which the
expert system is being applied. Desirable qualifications include an interest
in the project, availability, and good communications skills.
5.7 DESIGN
/
The design is important to the success of the expert system
application as it will be a guide for the Development Stage and the additional
prototyping which takes place there. There are two steps to the design
process. Each has a specific purpose.
5.7.1 One-page Design
The One-Page Design Document is a very high level view of the
system. It is completed at the beginning of the Definition and Design Phase
after preliminary knowledge acquisition has taken place. Its purpose is to
give a starting point to the design by giving brief description of each of the
components of the expert system. It should be emphasized that this document
5-13
-------
OSUER Directive *9028.00a
should be limited to one page, hence the name. This document is broken down
into the following six components :
o Overview of the system: This paragraph briefly states the
purpose and scope of the system. It also gives an estimate
of the measures of success of the system.
o Components and structure: A list of all components or
subdomains of knowledge and a brief description of how
each fits together in the overall structure.
o Interfaces: A description of any known interfaces
including any database and external interfaces. It also
states how the user interface will be established.
o External calls and data sources; An enumeration of all
external calls the expert system has to make and any data
sources which must be utilized by the expert system.
o Knowledge representation: A best approximation as to the
appropriate representation, based on what is currently
known about the problem domain. It also includes reasons
for the selection, because they may be useful later in
this phase and on into the Development Stage.
o Inferencing mechanism: The type of inferencing mechanism
is determined primarily from the knowledge representation
scheme. Refer to Section 4.9 to learn more about how to
select an inferencing mechanism.
5.7.2 Detailed Design
The Detailed Design Document further elaborates what was borne out
in the One-Page Design Document. In this document, the system is broken out by
expert system component or context. Within each component these questions
should be answered:
o What is the purpose of the component?
o What is the scope of the component?
o What information is the component going to need?
o Are there any specific external calls and/or data sources
needed?
5-14
-------
OSUER Directive #9028.OOa
5.8 PROTOTYPING STEPS x
The prototyping which began in the Concept Phase continues during
this phase with the refinement of the Proof-of-Concept Prototype and the
development of the demonstration Prototype.
i '
5.8.1 Proof-of-Concept Prototype
The proof-of-concept prototype is a small working system designed to
provide a preliminary feasibility assessment on the problem domain before
additional resources are allocated. This prototype is started in the Concept
Phase, but is refined and achieves maturity here. Ideally, this prototype is
designed to include only the top level features of the targeted system. For
example, the goal recommendations of the knowledge base can be represented
without the full line of reasoning to handle all possible cases in the problem
domain. The Detailed Design Document is merely alluded to and need not be
included at this stage.
5.8.2 Demonstration Prototype
The Demonstration Prototype is an extension of the Proof-of-Concept
Prototype. The Demonstration Prototype should be small and specialized, based
on a narrow subset of the overall problem domain. In contrast to the Proof-
of-Concept Prototype, an essential element is that the knowledge base is
designed to be deep in one area while maintaining breadth in the other areas.
In this prototype, more attention is paid to user interface features to make
the system attractive to management and to the end users. Also, -the necessary
data requirements are addressed at this stage to enable the assessment and
design of the external data integration.
5-15
-------
OSUER Directive *9028.00a
CHAPTER 6
DEVELOPMENT STAGE
6.0 INTRODUCTION
Tile Development Stage is a translation of the design into a working
system. This chapter describes several success factors needed during
development, the types of prototypes produced, and knowledge acquisition.
6.1 OBJECTIVES
There are several key objectives in the Development Stage (Fig.
6.1). One is to document key issues to be borne out during prototyping.
These issues are discussed for each prototype in Section 6.10. In the
Development Stage key issues for the Implementation Stage should be
documented. The problem description, functional description and knowledge
are refined during development of the prototypes. Now is the time to increase
product visibility. Also, maintaining management commitment is a key
objective of the Development Stage. Another objective is to codify the
knowledge base and maintain its accuracy through testing.
6.2 DECISIONS
There are several key decisions that must be made during the
Development Stage (Fig. 6.1). The first decision is how-to proceed given
current funding levels. Before each prototype is begun, two questions must be
resolved. The first is, "To what degree has knowledge acquisition taken
place?" The second question is, "Is the prototype addressing the problem
sufficiently to be fully developed?" User and management commitment are very
important to the success of the project so decisions must be made to satisfy
both user and managerial requirements. Another decision that must be
addressed is whether the developer has identified and resolved security and
backup issues. :
6.3 PRODUCTS ' .
There are several products associated with the Development Stage
(Fig. 6.1).
o Project management plan
o Test and validation plan
o Support plan
o Knowledge notebook
o Full Prototype
o Production System
o Development documentation
o Development decision paper.
6-1
-------
System UfeCyde
for
Expert Systems
OSWEP DIRECTIVE #902B.OOa
and
Design
Irnpiornerttaton
Objectives
Document key Issues to be borne out during prototyping
Document key decisions for the Implementation phase
. Perform Implementation planning before full Implementation
. Refine knowledge
Refine the problem description and functional description
. Obtain user acceptance
Increase product visibility
Maintain management commitment
. Codify knowledge
Maintain knowledge accuracy through testing
. Declsons to be sddressed Include:
how should development activities proceed given current
funding levels
Decisions - has the prototype addressed the problem sufficiently to be
Implemented
- to what degree has knowledge aqulsltlon .taken place
«has the project gained user and management commitment
° have security and backup Issues been Identified and resolved
New
Products
Project Management Plan
Test and Validation Plan
Support Plan
Knowledge Notebook
Full Prototype(s) (If needed)
Production System
Development Documentation
Development Decision Paper
Figure 6.1
Development Stage
Objectives, Decisions, and Products
-------
OSUER Directive #9028.OOa
6.3.1 Project Management Plan
The Project Management Plan is used to make sure the Development
Stage is run smoothly. It specifies who is in charge of each step of
development. It also specifies the resources each person has control over and
outlines the hierarchy of management when there are conflicts in the decision
making process.
6.3.2 Test and Validation Plan
The Test and Validation Plan is developed to specify how testing and
validation should be handled. It will contain a methodology for designing
tests for the expert systems, and it will list resources available during
testing as well as specifying when testing and validation will take place.
Key staff involved in the validation process are named at this time.
6.3.3 Support Plan ' .
The Support Plan contains information on what support services are
available. All resources and the time they are needed are specified in this
plan. Refer to Section 2.5 for information on what resources are necessary in
the Development Stage.
6.3.4 Knowledge Notebook
The Knowledge Notebook is used throughout the Development Stage for
knowledge acquisition and is kept by the primary knowledge engineer. Refer to
Section 6.9 for a description of this document.
6.3.5 Full prototype
The Full Prototype is an intermediate product in the Development
Stage. This prototype is described in Section 6.10.1.
6.3.6 Production System
i
The Production System is the primary.product of this stage. This
stage is not complete until the Production System is produced. The rest this
chapter outlines the steps necessary to produce it.
6.3.7 Development Documentation
The Development Documentation is produced while the prototypes are
being developed. It contains information on how each prototype was developed
and documentation on the knowledge representation and control structures. It
also contains the documentation for any supporting software of the expert
system.
6-2
-------
OSUER Directive #9028.OOa
6.3.8 Development Decision Paper
The Development Decision Paper is a documentation of all decisions
that were made during this stage. It lists each decision and the reasons and
facts supporting why the decision was made.
6.4 SUCCESS FACTORS
There are several areas that are crucial to the success of the
Development Stage. These factors are grouped below according to the area they
address. /
6.4.1 Knowledge Engineering
Knowledge engineering begins in the Concept Phase. The knowledge
engineers should devise a method to resolve conflicts among multiple sources
of expertise with differing specialties. Methods for dealing with conflicts
are given in Section 6.8.
It is wise to coordinate programmers and knowledge engineers, but
keep in mind that there may be organizational separation and differences in
background. Knowledge engineers should have a strong computer background to
facilitate communication with programmers. Knowledge engineering is most
effective when proven knowledge acquisition methods are used. These include :
o Unstructured interviews,
o Structured interviews,
o Observation,
o Interruption analysis,
o Constrained-processing tasks,
o Questionnaires, and
o Decision trees and decision tables.
' Many good techniques are available. The knowledge engineer should
apply one or several structured and unstructured techniques to document the
expert's [domain knowledge]. These techniques are discussed in detail in
Section 6.9.2.
The knowledge engineers and expert should think out all of the
ramifications of the rules on a periodic basis. There should be development
of knowledge throughout the project to allow for constant refinement and
experimentation. It is a good idea to periodically recheck the -accumulated
knowledge in order to better refine it.
Knowledge engineers need not be experts in the field; too much
domain knowledge has a risk of producing biases in the process toward the
knowledge engineers and away from users. Select knowledge engineers who are
familiar with the domain, but are not necessarily expert in it.
6-3
-------
OSUER Directive #9028.OOa
Knowledge engineering time estimates should be well thought out with
respect to reaching the appropriate depth of knowledge. There should be a
constant reminder to the project manager to plan for contingencies and expect
repeat sessions with experts and users.
6.4.2 Prototyping Methodologies
Below are several methods that contribute to a successful prototype.
o Use rapid prototyping with frequent interim deliverables
and decision points.
o Discard the prototype if a better design approach is
discovered. The purpose of a prototype is to firm up the
design specifications.
o The design should be modular to show all lines of
reasoning and specific functions the expert system is
required to perform. The prototype should also be
attractive as a demonstration vehicle to gain the support
of users and management. The development team should
accurately define and adhere to the system development
stages. This will allow the team to anticipate the long-
term impact of schedule deviations.
6'.4. 3 Validation Process
It is important to identify and validate external interfaces as
early as possible. This includes inputs and outputs, and other programs, and
algorithms - separately from the knowledge bases. Validation should be done
frequently and continually. There is a tendency to ignore validation until
the end of the Development Stage. Ideally, it should be done after the
addition of each rule, but later can be reduced to the end of each session.
Expert system validity relies heavily on the validity of accessed data.
Issues to be considered when validating include the knowledge base,
recommendations, justification, rationale, type of inferencing and incomplete
data.
I
It is not always possible to test all possible rule outcomes. This
is often the case for larger systems, e.g, +500 rules. This emphasizes the
importance of keeping a "trace" of each session and the need for software
maintenance in the operational phase.
6.4.4 Testing Procedures
Testing is the most important part of the Development Stage because
there may be potential conflicts in the knowledge base. A thorough analysis
should be performed on a routine basis throughout development and upon
completion of development. Remember to test all possible rule combinations to
identify conflicting or overlapping rules. Some ways to do this are :
6-4
-------
OSWER Directive #9028.OOa
o Random tests by beta users,
o Selected test cases run through the system, and
o Ad-hoc testing by the expert.
Make sure to adequately test critical components and any examples or
rules generated by inductive systems. Thoroughly retest entire knowledge base
when changes are made. The process of testing expert system shells is easier
than testing implementations written in LISP or PROLOG because the inference
engines have already been thoroughly tested in a shell. Remember not to
underestimate the proper level of testing. Allocate sufficient time and
resources to do the testing.
6.5 ACTIVITIES
Two background activities should be completed during this stage to
ensure a smooth development of the expert system. These activities include
the end users and management staff.
6.5.1 Demonstrations and Briefings to Users
Periodically, and throughout the Development Stage, conduct
demonstrations for the users. In addition, the user should play a primary
role in the development of the user interface in order to assure acceptance of
the expert system in its delivery environment. User involvement here will
ensure ongoing support-for the expert system application. Frequent
demonstrations of the system and milestone reports to management for the
expert system application are necessary for continued user involvement.
6.5.2 On-going Progress Reporting to Management
Management should be updated on a continual basis as to the progress
of the development effort. If time or cost overruns are expected, management
should be given a detailed report. Using detailed progress reports, it is
advisable to demonstrate to management that as the project changes and the
requirements are refined, the effort Continues to be manageable and important.
y
On-going management commitment is vital to the success of the expert
system. To ensure management commitment, detailed progress reports should be
given weekly or in an appropriate time frame for the project. Delays should
be attributed to discrete events and not to the expert system technology
itself. If proper feedback is given, management will be more confident that
the development effort is under control.
6-5
-------
OSWER Directive #9028.OOa
6.6 KNOWLEDGE ACQUISITION
Knowledge acquisition is the process of extracting and representing
the expert's knowledge in the form of a conceptual model. A point which is
sometimes overlooked in acquiring knowledge is the fact that there are usually
multiple solution paths to reaching a conclusion, based on alternative
hierarchies of assumptions. Expert systems must be designed to deal with
these types of uncertainties, which vary depending on the domain being
examined. Experts are not always explicitly aware of precise concepts and
representations of their knowledge. Documenting this knowledge requires
considerable patience and cooperation between the knowledge engineers and the
expert. In order to conceptualize the problem the expert and knowledge
engineer must agree on issues such as:
o What are the factors involved in decision making?
o Which inputs give the expert difficulty?
o What are some of the relationships between factors?
o What factors are missing?
o How accurate are the factors?
o What inferences does the expert make?
o How are hypotheses formed?
o How does the expert's knowledge evolve?
o What factors suggest particular goals or concepts?
o What are the solutions?
6.6.1 Knowledge Sources
Several different [knowledge sources] the knowledge1 engineer can
draw upon while studying the problem are outlined below. Sometimes conflicts
in data will emerge from different sources.
6.6.1.1 Background Data
The knowledge engineer must be familiar with the domain so as not to
overly burden the expert with questions that can be answered from background
reading. Background can come from various sources including regulations,
guidelines, textbooks, and deliverables or talking with the expert. Any
policies and/or procedures that are already outlined or documented can be a
valuable source of knowledge.
6.6.1.2 Cases
By analyzing specific cases, the knowledge engineer will be able to
determine what information is potentially important and how the factors are
interrelated. Cases also give the knowledge engineer an overall feel for how
the problems were solved in the past.
6-6
-------
OSUER Directive #9028.OOa
6.6.1.3 Test Data
Providing test data or posing hypothetical situations to the expert
is another way the knowledge engineer may extract information. This is done
by observing the way the expert solves the problem with data provided in a
structured test format. Test data can be derived from historical data
allowing dummy cases to be contrived for purposes of gaining knowledge.
6.7 COLLECTION METHODS
Given below are various methods for acquiring knowledge about the
domain. First though, are some general interviewing principles.
6.7.1 Interviewing Principles
Points to remember when interviewing an expert include the
following:
o The knowledge engineer should be specific when asking
questions. The expert may not have remembered rules or
concepts, and will find it difficult to recall them.
o The expert should be encouraged to provide the information
in a way which is most natural.
The expert should not be interrupted during unstructured
interviews. The aim is to get the expert talking.
Despite the fact that the expert will probably digress or
repeat things, interruptions should be kept to a minimum.
It is important for the knowledge engineer to record all
information collected during an interview and save it in
the Knowledge Notebook. It is not always clear which .
parts of the dialogue are important, even if the questions
are planned. The use of a tape recorder or video can be
very useful if the expert is comfortable having the
information acquired using these techniques.
The knowledge engineer should listen to the way the expert
uses knowledge. It is not just facts, theories, and
heuristic that are important. The knowledge engineer
should listen carefully to the way in which the expert
manipulates knowledge.
6-7
-------
OSWER Directive #9028.OOa
6.7.2 Knowledge Acquisition Methodologies
There are various proven knowledge acquisition techniques which
currently exist. A few are described below.
6.7.2.1 Unstructured Interview
An unstructured interview is basically a free form interview in
which the expert talks freely about the domain of expertise. The
goal is to become comfortable with the expert and to see how the
expert views the problem domain. The unstructured interview is not
very efficient, but this technique can be used to address several
important questions:
o Who are the experts?
o Is the problem scope suitable for an expert system?
o What are the needs of the end users?
6.7.2.2 Structured Interview
The structured interview is the most basic knowledge collection
technique used by the knowledge engineer. -It is generally used
after unstructured interviews are conducted. This method is
. applicable to all types of problems and with all types of experts.
After the knowledge engineer has met with the expert several times
and structured some of the knowledge, the knowledge engineer then
tailors the interview around gaps in the information. Interview
questions should center on the expert's knowledge of factors,
relationships, and inferences.
6.7.2.3 Observation
In some cases, the best way to discover how an expert makes a
judgment, diagnosis, or design decision is to watch the expert work.
This method is used primarily when the task at hand involves more
than just a cognitive process to achieve a goal. For instance,
determining which equations the expert refers to most often while
solving a problem. The knowledge engineer may discover the
knowledge in various ways.
One way is to watch, take notes and follow the expert's thinking
processes. The drawbacks to this are that the process is time
consuming and that the expert may perform differently while being
observed.
A second method is to record the session. This can be advantageous
because it:
6-8
-------
OSUER Directive «9028.00a
o Retains all information from the interview,
o Can keep pace with the expert, and
o Keeps the knowledge engineer's attention on the interview
and not on recording the information.
The drawbacks to this method are:
o All information must be transcribed,
o Irrelevant pieces must be filtered out,
o There is nothing to refer to during the interview, and
o The expert may be intimidated by a recording device.
6.7.2.4 Decision Matrix
Here the knowledge engineer forms a table or matrix with the
decisions to be made along one axis and the different factors that
go into making these decisions on the other. This method is used
when many factors are borne out during interviewing and must be
organized in some fashion. The purpose is to better diagram the
decision making process and to identify gaps in the knowledge for
future interviews.
6.7.2.5 Interruption Analysis
Interruption Analysis is especially good for validation of the
reasoning processes already encoded in the expert system
application. It works in conjunction with the observation technique
described in 6.7.2.3. In this method the knowledge engineer
observes while the expert proceeds to solve a problem without
verbalizing the process. When the process gets to the point that
the observer can no longer understand.the expert's actions, the
knowledge engineer interrupts. The knowledge engineer then probes
deeper, asking the expert about what was happening at that
particular instant. Note that once the process is interrupted there
is a chance that it cannot be started up again.
6.7.2.6 Constrained-Processing Tasks
In this method the expert is requested to solve the problem in a
much shortened time frame or with limited information. Experts are
.generally uncomfortable with limited information or time available
to solve the problem. Some experts would rather give no answer at
all than give one based on incomplete information. Explain to the
expert that this is not a test of their expertise but rather a means
6-9
-------
OSUER Directive #9028.OOa
of extracting additional information about the domain. Once the
expert opens up and becomes more comfortable about giving uncertain
or qualified judgments, much can be learned about the experts's
reasoning processes. There are two types of constrained processing
tasks.
The first is a constrained time task. Constrained time tasks limit
the time an expert has to solve the problem. When a time constraint
is imposed, the expert will generally take the most direct path to
the solution. This method is used to eliminate extraneous thought
processes which may be occurring without the expert realizing them.
The second is to limit the information available to the expert.
This is a good way to determine which factors or data are most
significant by forcing the expert to rely heavily on their knowledge
and reasoning skills. Formulation of hypothesis, use of heuristic,
and strategic thinking are important in limited information tasks.
Often, the expert is unaware of factors which are most important and
this is an excellent way of finding out.
6.8 RESOLVING CONFLICTS
As the knowledge engineer gains more and more knowledge about the
domain, conflicting.information builds. Listed below are some techniques used
to resolve these conflicts.
i
6.8.1 Focus Groups
When there are cases where multiple experts having different domains
of knowledge are involved, focus groups can be formed to decide on the best
method of resolving the conflict.
6.8.2 Delphi Method
The delphi method consists of writing out each piece of conflicting
information and evaluating each for its own merit without regard as to its
source. Rationale is given for each and an agreement is made as to the best
answer.
6.8.3 Consensus
A general consensus can be made based on discussions with multiple
experts. The disagreement may have been simply the way in which the knowledge
was represented or presented. This method is often best when dealing with
incomplete or inconsistent knowledge.
6-10
-------
OSUER Directive #9028.00«
6.8.4 Quality Assurance Committee
The quality assurance committee is set up specifically to resolve
conflicts and maintain knowledge integrity. This committee should be set up
initially if many conflicts are expected to occur among experts.
6.8.5 Hierarchically
Though not the ideal way, sometimes it is more efficient to resolve
conflicts in a hierarchical fashion. If people at one level have trouble
resolving a conflict the person higher up in the chain of command such as a
department chief or manager will resolve it. This method works best early in
the Development Stage where decisions must be made quickly to avoid
unnecessary delays. If the manager being asked to make the decision is not a
domain expert, reconsider escalating the issue to that level.
6.9 KNOWLEDGE NOTEBOOK^
The Knowledge Notebook contains notes on all knowledge engineering
sessions with the expert including any knowledge maps the knowledge engineer
has developed and any real or perceived knowledge gaps. The Knowledge
Notebook serves to organize the knowledge engineer knowledge and prevent the
knowledge engineer from asking the expert redundant information.
6.10 PROTOTYPING STEPS
Prototyping reaches its final stages during the Development Stage.
Below are descriptions and development methodologies of the Full Prototype and
the Production System produced during the Development Stage.
6.10.1 Full Prototype
The Full Prototype contains most of the knowledge and the
representation and control structures have been determined. Sample cases and
lines of reasoning are expanded upon. Difficult cases are added and any
external interfaces are connected. Testing and verification now play a more
important role as information and lines of reasoning are expanded.
The purpose of the Full Prototype is to verify the problem domain,
the knowledge representation schemes and control structures, data
requirements, and interfaces.. If major revisions are needed either to the
knowledge representation or control strategies, a new approach can be designed
.following the guidelines in CHAPTER 4.
Prototyping is an exploratory process. If at this point the design
is no longer valid, the prototype can be reevaluated based on the new
information and redesigned if appropriate. The efforts that went into the
6-11
-------
OSWER Directive #9028.OOa
development are not lost; they represent the design phase in conventional
systems.
6.10.2 Production System
This is the actual system that will go out into the field. At this
point development is complete and the expert system has a finished knowledge -
base which has been compiled and is secure. Any additions or deletions which
must be performed as a result of further evaluation and testing are
implemented. All user interfaces and integration with external interfaces are
in place. In addition, the system has been fully tested and validated. Any
disclaimers about the use and liability of the expert system are added to this
prototype. The completed system should be robust and user ready.
6.11 TESTING AND VALIDATION
Prior to the field testing performed in the Implementation Stage
(see Section 7.6.2), it is important that the expert system is tested and
validated by the developers. This will ensure that the Production System is
working effectively. The items to be tested include the data, the knowledge
base, and the flow of logic.
6.11.1 Data
Data validation is the most often neglected task in expert system
development. Databases, external files, and tables must be correct before the
expert can reason using this data.* If the data is incorrect the system will
fail to produce the required results. This is true of the results to date as
well as any future results.
6.11.2 Knowledge Base
The data that results from the knowledge acquisition processes
should be comprised only of valid information. The data should contain truths
about the expert's knowledge and reasoning and the expert's perceptions and
observations on the domain. Also, the data must be complete in the sense that
it covers all of the relevant subdomains of knowledge.
The knowledge should accurately reflect the way the expert views the
domain and reasons about the task at hand. One way to check this is to see if
there is a consensus across experts. Remember that some people feel a bias
toward a computer generated result and may grade the response more harshly
than one would an expert. To get around this, intermix the computer results
with those of an expert panel without differentiating between the two.
6-12
-------
OSUER Directive 09028.OOa
6.11.3 Logic Flow
There are various methods used to validate the logic flow.
o Run test cases using sample problems to see if the expert
system arrives at the same conclusion the expert does.
o Follow chains of reasoning, picking one specific topic
within the domain and following the reasoning process the
expert system follows to see if it matches the reasoning
followed by the expert.
o Change the firing sequence to see if the expert system
produces the same results if the firing sequence is
changed. If the system fails to respond in the same
manner, then the result was a artifact of the control
structure and not of the knowledge.
6-13
-------
OSUER Directive *9028.00a
CHAPTER 7
IMPLEMENTATION STAGE
7.0 INTRODUCTION
The Implementation Stage is the second part of the Development and
Implementation Phase. It is the last step before the system is made fully
operational to all of the users in the field. At this point, the prototype
system has been tested by the expert and the developer, and revisions have
been made. The system, in its nearly final state, is ready to be tested in
the field by the actual users through the beta test. The Implementation Stage
includes four main steps:
*
o Testing and evaluation of the expert system by beta test
participants
o Revising the expert system by knowledge engineers
o Registering users and distributing the system
o Training users and providing System Documentation.
7.1 OBJECTIVES
The objectives of the Implementation Stage (Fig. 7.1) are mainly to
test and finalize the expert system and prepare it for distribution. This
includes a beta test of the system by potential users and a revision of the
system following the test. In preparation for distribution of the system, a
training program and documentation are completed, and users are registered.
The expert system itself is also prepared for distribution through final
debugging and copies of the run-time version are produced.
7.2 DECISIONS
A number of important decisions are made during the Implementation
Stage (Fig-. 7.1). These involve the testing and revision of the system, as
well'as the distribution of the system and training of users.' Decisions
involving the beta test include determining who will participate in the test,
and how long the test will last. The group of beta users should be kept to a
relatively small number in order to limit the volume of comments on the
system. However, the beta users should comprise a representative sampling of
the types of, potential users of the system. An intensive beta test should
generally last 2-3 weeks, or longer if necessary. This will give the beta
users adequate time to learn how to operate the system, apply it to their
work, and evaluate it.
After beta users comments have been received, the developers of the
expert system must decide what revisions they will make. This will involve
reviewing the comments and determining how critical the suggested changes are
7-1
-------
OSWER DIRECTIVE #9028.00a
Deafen
Objectives
Have beta user* test expert ayatem In the field
Revise the expert ayatem If necessary
Prepare training and documentation for the ayatem
Prepare ayatem for full distribution
Decisions to be addressed Include:
- who will test the expert ayatem
how long will the beta teat laat
Decisions *wnat revisions will be made to the ayatem
la the expert ayatem ready for distribution
- how will training be Implemented
la sufficient funding available for the remainder
of the life cycle
New
Products *
Llat of Beta Teat Participants
Compilation of
Beta User Comments
Scope of User Community
Documentation of Expert .System
Distribution Plan and Schedule
Implementation Decision Paper
Figure 7.1
Implementation Stage
Objectives, Decisions, and Products
-------
OSUER Directive #9028.OOa
to the success of the expert system. Project management must agree to the
developer's recommendations of what revisions to make. There is an important
trade-off between achieving functionality in the system and using resources
efficiently, and this should be considered when determining which revisions
are the most important. As revisions are implemented, the developers will
decide when the system is ready for final distribution.
Decisions involving the implementation of a training program are
also made at this time. Training options are discussed in Section 7.6.4. As
this stage in the life cycle comes to a close, management staff must determine
that there is sufficient funding for the remaining phases.
7.3 PRODUCTS
The Implementation Stage results in six products (Fig. 7.1). These
include:
o Distribution Plan and Schedule
o Implementation Decision Paper
o Scope of User Community
o List of Beta Test Participants
o System Documentation
o Beta User Comments
7.3.1 Distribution Plan and Schedule
At the beginning of this stage, it is important to draw up a
Distribution Plan and Schedule, summarizing the distribution activities to be
completed. Following the schedule closely is especially critical in this
stage because of the increased contact with people outside of the system
development staff.
7.3.2 Implementation Decision Paper
* >
The Implementation Decision Paper should be composed early in this
stage. It should include all of the decisions to be made during this stage
(see Section 7.2), and who is responsible for carrying them out. The
Implementation Decision Paper should be approved by the project management
staff to ensure that they are aware of the decisions and schedule that must be
carried out.
7.3.3 Scope of the User Community
The final Scope of the User Community must be established at this
time in order to plan for how many systems will be distributed and to provide
adequate support once the system is shipped to the field.
7-2
-------
OSUER Directive #9028.OOa
7.3.4 List of Beta Test Participants
Potential beta test participants are first selected from the group
of end users. Beta test participants should include those who are interested
in the development of the system and have the time to effectively test the
system. After contacting the potential beta test participants and locating
those willing to be involved, a List of Beta Test Participants will be
compiled.
7.3.5 System Documentation
Documentation of the expert system and instructions for its use are
written during this stage. It is imp.ortant that these documents are
completed, at least in draft form, prior to the beta test, in order that they
can also be evaluated. Further requirements for System Documentation are
covered in Sections 7.4.1 and 7.6.5.
7.3.6 Beta User Comments
Following the beta test, the beta test participants will be
contacted and their comments on the system will be solicited. These comments
will be integrated and compiled for .use by those involved in the revision of
the expert system.
7.4 SUCCESS FACTORS
There are several factors in the Implementation Stage that must be
considered in order to ensure the success of the expert system. Most of the
factors involve the transfer of the system to users in the field and the
initial use of the system. In order to produce a successful expert system, it
is important to:
o Provide useful, readable documentation
o Provide sufficient and organized training
o Provide sufficient technical support
o Ensure that hardware in the field is adequate to support
the expert system software
o Avoid problems with licensing and run-time versions
o Maintain management commitment during the distribution of
whe system.
Most of these factors can be covered through planning and
coordination prior to this stage.
7-3
-------
OSUER Directive «9028.00a
7.4.1 Documentation
Good documentation can be produced through a quality assurance
system, requiring reviews by the developers of the system, as well as
management. A draft of the documentation must be provided to the beta users
for their comments on its clarity and comprehensiveness. For more information
on expert system documentation see Section 7.6.5.
7.4.2 Training
The planning of the training process should be supervised by
'management, who will work with the developers and the trainers. For more
information on training issues, see Section 7.6.4.
«
7.4.3 Technical Support
In order to provide sufficient technical support, a technical
support team should be organized before the. system is distributed. Their
duties should include manning a support hotline and providing on-line help to
the users. The technical support team should include individuals who were
involved in developing the expert system, writing the documentation, and
providing training sessions. More information on technical support for the
expert system can be found in Section 8.5.2.2.
7.4.4 Hardware Issues .
Problems with hardware in the field can be avoided if the users are
given ample warning of what equipment will be required to run the expert
system. This process is explained in Section 7.6.6.
7.4.5 Licensing Issues ,
Licensing and run-time issues should have been confronted early in
the life cycle (s'ee Section 2.5.6.1). During the Implementation Stage, it
should be necessary only to confirm with the vendor of the shell what steps
should be taken to distribute run-time copies of the expert system. The
matter of who will pay for the run-time copies must also be considered at this
time. For more information on run-time and licensing issues, see Section
7.6.8.
7.4.6 Role of Management
Management commitment during this stage can be ensured if, early in
the life cycle, management is made aware of the fact that they will play a
large role in the Implementation Stage. They should know that they will be
responsible for the scheduling and planning involved in this stage., as well as
the actual distribution of the system and all of the issues accompanying the
distribution process.
7-4
-------
OSUER Directive «9028.00a '
7.5 ACTIVITIES
There are several important activities in this stage that lead up to
the distribution of the system. These include the collection of data,
testing, demonstrations, and management activities.
7.5.1 Data Collection
Collection and organization of data is important during this stage.
Sufficient staff time should be allocated for the following tasks:
o Contacting possible users and recording their level of
interest .in participating in the beta test
o Selecting actual beta users and recording their names and
addresses on a mailing list
o Seeing that the beta test system and any accompanying
documentation is sent to all of the beta users
o Developing a questionnaire on the system and sending it to
all of the beta users or conducting the survey by phone
o Collecting the questionnaires and compiling responses and
suggestions from the beta users
o Registering all users of the system and maintaining the
registry.
7.5.2 Testing
The main testing to be conducted during this stage will be the beta
test. Resources should be allocated for the following tasks related to the
beta test:
o Contacting beta users (see Section 7.5.1) before and after
the test
o Technical support for both hardware and software
o Documentation/training
o. Revisions to the system following the beta test.
7.5.3 Management Support
Management plays an important role in the Implementation Stage, so
it may be advisable to allocate a staff member to serve as an assistant to the
manager. This person would provide administrative support for:
7-5
-------
OSUER Directive 19028.OOa
o The beta test,
o Data collection,
o Training and documentation,
o Licensing and run-time arrangements,
o Distribution plans, and
o Configuration management.
7.5.4 Demonstrations and Briefines for the Users
Contact with the users is especially important during the
Implementation Stage, as the expert system is nearing its final state. The
users will be able to see the actual system they will be working with, rather
than a prototype. Demonstrations and briefings involving possible users can
be useful in determining who is interested in participating in the beta test
and final evaluation of the system. They will also determine the overall
level of interest in the system. This information can be used to define the
size of the system, and to allocate sufficient staff time for distribution,
technical support, and maintenance of the system.
Demonstrations and briefings can also be used as training sessions
for both the beta users and initial system users. In some cases, the
developers of the system may choose to meet with some of the beta users after
the test to receive their comments on the system. This would allow for a more
interactive evaluation than simply filling out a questionnaire.
7.5.5 On-going Progress Reporting to Management
Management should receive a complete schedule of all tasks involved
in implementation at the beginning of the stage. They should immediately be
informed of any changes in the schedule, because many aspects of system
implementation involve outside parties, and maintaining good public relations
is very important. Progress reports should include updates on:
o Beta test,
o Distribution plans, and
o Any major problems encountered.
7.6 DISTRIBUTION STRATEGY
An important part of the Implementation Stage is planning and
scheduling the distribution of the system. This stage of implementation has a
number of parts and should include close management involvement.
7.6.1 Rollout Plans and Schedule
At the beginning of the Implementation Stage, all tasks to be
performed must be carefully planned and scheduled. Specific dates should be
set for the beginning and end of the beta test, and for the distribution of
7-6'
-------
OSUER Directive #9028.OOa
the final system. All aspects of the stage should be planned carefully, and
sufficient staff and time should be allocated for each task. It is very
important that the beta test and distribution of the final system are carried
out smoothly and that the schedule is closely followed, because this stage
will be receiving a great deal of external attention, and public relations is
an important factor.
7.6.2 Beta Test
The beta test is a very important step in the life cycle of an
expert system. During this step, the system will be distributed to a few
selected participants who have agreed to try the system out for a defined
period of time, then provide their comments on the system and any suggestions
they have for changes or improvements. The aspects of the system to be tested
include:
o Hardware,
o Software,
o Support procedures,
o Documentation, and
o Training.
The beta test provides an opportunity for the system to be tested on
[real-world problems] by the people who will be using the system. They will
know best if the system will be effective and assist them in their work...
7.6.2.1 Contact Beta Users
Potential beta users should be contacted early in the Implementation
Stage to determine their interest in participating in the test. A
small number of interested parties will be selected to participate
in the beta test. These participants will receive the entire expert
system, along with draft documentation on the system. The beta
users can also be provided with training on the use of the system,
if necessary. This would provide an opportunity for the training
materials to be tested, as well.
7.6.2.2 Register Beta Users
The beta users should be registered before they begin testing the
software. This will make it easier to identify unregistered beta
users and explain to them the risks of using beta test software.
The distributors' risk and exposure will then be limited if the
unregistered persons decide to use the system anyway.
7.6.2.3 Collect Comments from Beta Users
After approximately 2-3 weeks, or longer if necessary, the beta
users will be required to answer some questions about the expert
system and provide comments and suggestions for changes or
7-7
-------
OSUER Directive #9028.OOa
improvements. The beta users may be sent a questionnaire, which
they will have to complete and return, or they may be contacted by
phone and asked the questions directly. The developers may also
wish to meet with some or all of the beta users and conduct an open
forum to discuss the expert system.
7.6.2.4 Revise Expert System
After the beta users responses and comments have been compiled and
evaluated, the developers will revise the expert system accordingly
and prepare a final version for distribution to all users.
7.6.3 Registered Users
""^"""^^"^"^" - T-J ~"~^^ »
Registration involves keeping a list of all authorized system users
and assigning each with a registration number. This process will be
supervised by the management support staff. .All users of the final expert
system should be registered. However, if there will only be a small number of
users, this may not be necessary.
Registration ensures that all of the users are using the official
run-time version and that unauthorized users will not be supported. Each user
will be assigned a registration number, which will be activated when the user
returns the registration card included with the software. The registration
number must be stated before any information or technical support will be
provided to the user.
A list" of all registered users and their addresses and phone numbers
will be maintained so that they can be informed of any errors in the system or
documentation, and be provided with new versions of the system when they are
developed (see Section 7.6.9).
7.6.4 Training Issues
Training of all system users is an important part: of ensuring that
the expert system is operated and maintained properly in the field. Users can
be trained directly through face-to-face training sessions,, or indirectly
through other training methods.
7.6.4.1 Direct Training
Direct training of the users can reduce the amount of technical
support necessary and can make the transition into the initial
operation of the expert system much smoother. However, direct
training may not be necessary or desirable, if, for example:
o The users are widely scattered geographically,
o There is a large number of users,
7-8
-------
OSUER Directive 09028.OOa
o The system is self-explanatory, and/or
o The majority of the users have seen demonstrations of the
expert system and have a good understanding of its
operation.
If direct training is implemented, those performing the training
sessions should be persons who were directly involved in the
development process, such as the knowledge engineers, or instructors
who have been trained directly by them. Training sessions could be
at the users' location, or in a central location, with one or more
sessions.
It should be determined if every user will be involved in the
training, or if there will be a single representative or small group
from each area or region. Training a small number of
representatives is known as a "train-the-trainers" approach. After
these individuals have been trained, they will be able to train the
other users in their own area or region.
7.6.4.2 Other Training Methods
If direct training is 'not implemented, several other methods of
training can be used. Training can be put on-line as part of the
expert system, it can be recorded on videotape and sent to the
users, or it could be provided in the form of a written training
manual.
It may be useful to provide training at several levels. The initial
set of training could be a beginning level, and could be followed by
more advanced training, after users have some experience with the
system.
7.6.4.3 Intended Purpose of the Expert System
An 'important part of the training will be emphasizing the intended
use of the expert system. An expert system is meant to be advisory
in nature, and will not take the place of a human. The users will
receive recommendations from the expert system, but it remains their
responsibility to make all final decisions, either accepting or
rejecting the system's recommendations.
7.6.5 Documentation
Document, tion of the expert system is very important and is
something that is often inadequate. The more clear and easily understandable
the documentation is, the less technical support will be needed. The
documentation should probably not be written by someone who was directly
involved in the development of the expert system, such as a knowledge engineer
7-9
-------
OSUER Directive #9028.OOa
or programmer. They may leave out details that are obvious to them but not to
a first time user.
The developers of the system should be involved in a review of the
documentation, but it may be preferable to have the documentation written by
someone who is trained by the developers, but who has Just become familiar
with the system. Such a person will be more aware of every necessary step and
detail involved in running the system. Another option would be to use a
technical writer, who could study the system and be briefed by its developers,
then write the documentation.
The documentation should be brief, clear, concise, and use graphics
and pictures of the actual screens whenever possible. No system will be
shipped without adequate documentation.
7.6.6 Hardware Requirements
Prior to distribution of the final expert system, all users will
need to have the hardware necessary to support the expert system. Early in
the Implementation Stage, all potential users should be informed of the
hardware, equipment, and facilities they will need to run the system, such as:
o Type of computer, amount of memory, whether a hard disk is
necessary
o Type of furniture and work space needed to accommodate the
computer
o Type of printers
o Phone lines and modems
o Electrical hookups and surge suppressor.
Ideally, the hardware selection process, described in Section 4.11
took into account the facilities of the majority of the potential users, and
the target hardware was chosen accordingly.
7.6.7 Software Requirements
Prior to distribution of the final expert system, the software must
be prepared for the users. A run-time copy must be prepared for each user and
labeled. Everything necessary for the system should be packaged together and
sent to all users with detailed instructions.
7.6.8 Run-time and Licensing Issues
Before the final expert system is distributed, all run-time and
Licensing issues must be resolved. Although the major decision point on this
subject occurred when the development and delivery environments were evaluated
7-10
-------
OSUER Directive #9028.00a
in the Definition and Design Phase (Section 5.5), these issues must be
addressed in the Implementation Stage. All user facilities will receive a
run-time version of the software. Some of the expert system shells currently
on the market require a fee for each run-time copy issued, while others have a
one time fee that covers all run-time copies. If the run-time fees are to be
allocated to the users, a system for the collection of fees must be
implemented.
Some expert system shells do not include run-time modules. These
shells do not provide any protection for the expert system, and would allow
the users to make changes to it. If this is the case, a disclaimer must be
included, stating that the system will no longer be valid if it is tampered
with or changed in any way.
The final version of the expert system should include the copyright
information for the company that owns the rights to the expert system shell,
as well as a notification that it is illegal to make copies of the software.
7.6.9 Configuration Management and Version Control
When the expert system is revised or a new version is developed, it
will be necessary to distribute it to all users. Users must have the
currently valid version, and only that version, to avoid confusion and legal
infractions of the licensing agreement. When a new version becomes available,
all users will be notified. They will be required to send in their old
version, which is marked with their registration number, in order to receive
the new one. Other matters to be considered when the system is revised
include:
o Changes or addendum to the documentation
o Retraining
o Removal or overwriting of old versions from all of the
users' hard disks. . :
7-11
-------
OSUER Directive 49028.OOa
CHAPTER 8
OPERATION PHASE
8.0 INTRODUCTION
After completing all four major phases (Initiation, Concept,
Definition and Design, and Development and Implementation), Operation is the
last of the five major phases in the OSWER system life cycle. This phase is
divided into three stages, Production, Evaluation, and Archive.
The Production Stage starts with the results of the work conducted
in all prior stages, and activities of this stage are frequently in response
to those results. In other words, deficiencies and problems not resolved
adequately in prior stages are usually identified by users during the
Production Stage.
The Evaluation Stage differs from .other phases and stages of the
system life cycle in that it occurs simultaneously with another stage
(Production) and may be repeated more than once during the system's
operational life. During this stage, the system is reviewed formally to
determine whether it is operating correctly and efficiently from a technical
standpoint, and whether it continues to effectively address the information
management problem.
The Archive Stage preserves information not only about the current
production system, but also about the evolution of the system through its life
cycle.
8.1 OBJECTIVES
Figure 8.1 graphically illustrates the objectives associated with
the Operation Phase. Each stage's objectives are cited below.
8.1.1 Production
The first objective of the Production Stage is to use the
capabilities of the system to solve the information management problem by
delivering the system to the users. Proper use of the system will require
user training in the special capabilities and limitations of expert systems.
The second objective is to identify potential changes needed to
ensure that the system and data continue to solve the information management
problem. Changes may take the form of routine maintenance or may constitute
enhancements to the system or databases. Careful management of the knowledge
base, including revalidation after each change, will be required to ensure
continued satisfactory performance.
8-1
-------
OSWER DIRECTIVE #9028.000
Objectives
Decisions
Production:
deliver the system to the users
Identify potentlsl changes to the system
- develop snd Implement msintensnee changes end minor
enhancements
Evaluation:
determine the extent to which the system effectively snd
efficiently addresses the Informstlon msnsgement problem
»determine whether the' system should be continued,
enhanced, or replaced/archived
Archive:
- end the operation of the system In sn orderly msnner
- ensure system components snd dsts sre properly archived
or Incorporated into other systems
Production approsch decisions include:
what evaluations of the system snd dsta should be conducted
- what new or additional user' support activities are needed
Production execution decisions Include:
- what changes or enhancements are needed
- should sn enhancement be made within this stage or
given Ita own life cycle
Evaluation approach decisions Include:
how well does the expert system solve the Information
management problem
New Performance Report
Products Post Implementation Evaluation
Report
System Evaluation Report
System Disposition Report
Figure 8.1
Operation Phase
Objectives, Decisions, and Products
-------
OSWER Directive #9028.00«
The third objective is to develop and implement maintenance changes
and minor enhancements. All maintenance and minor changes are controlled
through baselines, particularly in the case of expert systems.
i>
8.1.2 Evaluation- #
i;
The first objective of the Evaluation Stage is to determine the
extent to which the system effectively and efficiently addresses the
information management problem. Efficient operation, effective management,
and current functional and data requirements are all considered in this stage.
The second objective is to determine whether the system should be
continued, enhanced or replaced/archived. This decision requires appraisal of
advances in technology and estimation of cost/benefit to upgrade the system.
8.1.3 Archive
The first objective of "the Archive'Stage is to end the operation of
the system in a planned orderly manner. Care must be taken not to disrupt
OSWER programs that use the system as well as other systems that use the data
and/or software of the current system.
The second objective is to ensure that system components and data
are properly archived or incorporated into other, systems. Resources dedicated
to the system-being deactivated should be reallocated to functioning systems
where appropriate. Data and software that- are not currently needed by other
systems should be archived against future needs.
8.2 DECISIONS
Many of the decisions made during this phase of expert system
development are identical to those made for conventional systems. Only
special expert system development considerations are cited below. If
additional detail is required, refer to the LCM Guidance.
8.2.1 Production
8.2.1.1 Approach Decisions
The Production Stage deals with two approach decisions. First, what
evaluations of the system and data should be conducted? Many
factors may affect the selection of specific evaluations including
new requirements, system performance, new technology, and cost
experience (i.e., costs accrued over the development life cycle).
Because expert system technology in particular is subject to rapid
evolution, system developers must stay abreast of advances in AI
tools and techniques.
8-2
-------
OSUER Directive *W28.00a
Second, what new or additional user support activities are needed?
Much will be learned about the required level of system maintenance .
and user training as the system is developed. Because improper use
of an expert system could have serious repercussions, these topics
should be monitored.carefully to ensure that the systeia is used
properly. ;;
8.2.1.2 Execution Decisions
There are two execution decisions. First, it must be determined
what changes and/or enhancements to the system and database are
needed.
Second, should a particular enhancement be implemented within this
stage, or given its own life cycle? Potential factors to be
considered which tend to require the start of a new life cycle
include:
o The processing of additional information,
o Major impacts on the system,
o High cost of accomplishing the change, or
o Significant impact on multiple OSVER, regional, or state
offices.
Review and approval thresholds will play a major role in determining
what action to take. The capacity of the expert system software to
be modified will influence this decision.
8.2.2 Evaluation
The Evaluation Stage deals with what changes or improvements in
system and data functionality, quality, and/or performance are
needed. This approach decision made during this stage will
determine how well the system solves the information management
problem. Information access, processing, and content are all
considered. Expert systems will require special attention in
evaluating changes to the knowledge embedded within the system
(e.g., as regulations or allowance -changes).
8.3 SUCCESS FACTORS
Several factors that can impede success it not considered in the
Production, Evaluation, and Archive Stages are described below. This Section
focuses on perceptions of what expert systems can and cannot do and offers
solutions or alternatives.
8-3
-------
OSWER Directive #9028.OOa
8.3.1 Production
In the Production Stage, users should not become overly reliant on
an expert system's recommendations. Because the advice offered by the system
is so similar to the human expert's recommendations, the advice may be
accepted as infallible. This is particularly dangerous when the system is
working with incomplete or incorrect information. Users should maintain an
attitude of skepticism in proportion to the consequences of the system
rendering incorrect advice.
8.3.2 Evaluation
In the Evaluation Stage, the organization should be cognizant of
changes that affect the system's performance and should not treat them too
casually. Relevant changes such as development of new techniques or
expertise, and enhancements in hardware and software may all affect the
system. Formal evaluations of all perceived changes and their impact on the
system are required.
Translating changes into modifications to the knowledge base will
require an iteration of.knowledge engineering with the expert. Depending on
the extent of the changes, this iteration could almost be a mini-project
development effort.
Over time, the problem that the system was developed to address may
cease to exist or may be subsumed in a larger problem addressed by another
system. "Pride of ownership" can inhibit shutting down a system that is no
longer required. This is obviously wasteful and inefficient. All programs
and systems should be viewed objectively.
8.3.3 Archive
In the Archive Stage, it is important to retain components of the
system that are useful at a later date. In general, knowledge is always
valuable. Knowledge in automated form is easily archived and retrieved as
needed. However, it is also easy to overcompensate for the first by retaining
everything. If an information management problem ceases to exist, significant
portions of the solution to the problem are probably discardable. Large
systems may consume considerable physical and logical storage space; both are
expensive and should be used efficiently.
8.4 PRODUCTS
Figure 8.1 also graphically illustrates the new products associated
with the Operation Phase. They are similar to the ones cited in the LCM
Guidance. Each stage's products are listed below.
8-4
-------
OSUER Directive #9028.OOa
8.4.1 Performance Report
The Performance Report describes the experience of system, knowledge
base, and database users during Production, noting unanticipated events and
potential problems. This report serves as a diagnostic ;;ool to aid the
project manager, as well as assisting evaluations of the;system, knowledge
base and database(s). This report is usually brief, and may include extracts
from computer facility reports that identify the resources used by the system
and database(s).
8.4.2 Evaluation
8.4.2.1 Post-Implementation Evaluation Report
The Post-Implementation Evaluation Report presents a complete
assessment of the implemented system based on the experience of the
initial period of system operation. This report addresses all
facets of the system, including degree of satisfaction of functional
and data requirements, technical performance, and system management.
It also identifies potential new requirements not addressed by the
system. Specific recommendations are provided, where appropriate,
to help ensure that the system continues to respond to the
identified information management problem. With regard to expert
systems, particular attention should be paid to new or changing
functional and data requirements, ongoing training, and database and
knowledge base management.
8.4.2.2 System Evaluation Report
The System Evaluation Report presents the results of a formal
assessment of the system. 'The assessment may vary in scope,
focusing on how well the system addresses the information management
problem, technical performance of the system, and/or system
management practices. The report provides specific recommendations
where appropriate and notes those recommendations approved by
program management for action. If the evaluation is conducted by a
completely independent third party, the evaluation should include a
Section including the opinion of the project team with regard to the
findings and recommendations of the evaluation.
8.4.3 System Disposition Report
The System Disposition Report:
o Describes the rationale for ceasing system operations,
o Documents the plan for ceasing operations and effectively
archiving the various components of the system, and
8-5
-------
OSWER Directive #9028.OOa
o Provides information about the location of archived
materials.
This report is vital to ensure that information about the system can
be accessed to support reactivation of the system, or future reuse of portions
of the current system by other systems.
8.5 ACTIVITIES
In addition to activities resulting in the creation of the products
listed in Section 8.4, the project management and development team should
conduct activities to resolve human resource issues, ensure the successful
completion of the production stage, maintain the system, and evaluate the
system. These activities are described in the following Sections.
8.5.1 Human Resource Activities
8.5.1.1 Soliciting Feedback from the Users
The users of the expert system play the most important role in the
Production and Evaluation Stages. By this time the expert system is
a fully working, usable system and the user can thoroughly critique
its performance. They are best able to determine if the system is
efficient and effective.. It is important to gather the user's
comments and recommendations at this phase to evaluate if the system
should be continued,-enhanced, or archived.
8.5.1.2 Two-Wav Communication with Management
Management also has a heavy involvement in these stages. If the
users desire to continue using the system, they must have
management's support to allocate funding and resources. Likewise,
management must evaluate the user's requests and comments and
incorporate them so that the system maintains its effectiveness.
8.5.2 Production Activities
/
Project managers and software developers must also consider
distribution and end-user support for the completed system. The
distribution issues are presented in Section 8.5.2.1; the end-user
support issues in Section 8.5.2.2.
8.5.2.1 Distribution
8.5.2.1.1 Configuration Management
Formal review of requested changes to the system before
they are made is essential to the integrity of the system.
8-6
-------
OSWER Directive #9023.00*
The procedures for reviewing requested changes are
contained in the Configuration Management Flan (a part of .
the overall Froj ect Management Plan).
8.5.2.1.2 Software Disclaimers
\>
Software disclaimers use limited warranties to restrict
vendor and manufacturer liability to replacement of faulty
software. They typically deny responsibility for the
consequences of improper use of the software or for use of
outdated versions. Because the advisory capacity of
expert systems presents new implications for improper
system development or use, any software disclaimers by
vendors, manufacturers, or developers should be carefully
studied. Standard disclaimers will probably emerge in the
near future as expert systems applications become "off-
the-shelf" products.
8.5.2.2 End-User Support Requirements
8.5.2.2.1 Ongoing Training
Providing ongoing training is important in the Production
Phase. New users will require training to effectively use
the system, and experienced users will require retraining
whenever significant changes are made to the system.
Systems that generate critical output may require regular
certification of all users.
8.5.2.2.2 Documentation
i
All changes should be completely documented to provide
system users and those responsible for operating and
maintaining the system the information they need to
properly use the system and to.perform -other activities
for effective system use.
t
Effective reactivation of the system in the future will
depend heavily on having complete documentation. It is
generally advisable to 'archive all documentation,
including the life cycle products generated during the
earliest stages of the life cycle, as well as the
documentation for users and for operation and maintenance
personnel.
8.5.2.2.3 User Groups
Manufacturers of larger expert system shells support user
groups in their major client bases. These groups provide
8-7
-------
OSUER Directive #9028.OOa
a forum for exchanging experiences with the tool and with
particular applications of it. All organizations should
be represented at user group meetings.
8.5.2.2.4 User Feedback
User feedback helps to determine what enhancements to the
system are needed to continue to solve the information
management problem, including requested changes deferred
from prior stages, and developing and implementing these
enhancements.
8.5.2.2.5 Software Updates and Upgrades
Software updates and upgrades are necessary in the
Production Phase to ensure a successful evaluation for the
continued use of the system. Any new releases of system
software and applications software packages to operate the
system should be provided.
8.5.3 Evaluation Activities
Evaluation considerations that must be considered by the project
manager or software developer are maintenance for the system and a post
implementation evaluation.. These two considerations are described in the
following Sections.
8.5.3.1 Maintenance
During the Evaluation Stage, potential changes to the system are
considered to ensure that the system continues to solve the
information management problem. Changes may take the form of
routine maintenance or may constitute formal enhancements to the
system or databases.
1 8.5.3.1.1 Maintenance and Revalidation Options
There are three major sources of change proposals and
maintenance activities: users, experts, and the MIS
department.
o Users typically request minor enhancements that
provide additional functionality and/or improve
performance but do not alter the knowledge or
data structures. They provide maintenance
support by putting any changes through "real
world" testing. Experienced'user criticisms and
comments should be considered carefully: they
know best how the system works and does not work.
8-8 .
-------
OSUER Directive *9028.00a
o After an expert system has been developed and
fielded, the problem domain may change for
technical, regulatory, or procedural reasons.
The experts wj11 probably be the first to
recognize the impact of these changes on the
system. They should therefore be regularly
involved in assessing how well the system
continues to address the problem. They will also
assist in revalidating the system after any
change is implemented by evaluating the effects
of the change on other parts of the system.
o In most organizations, the ADF department is
responsible for actually implementing changes to
supported systems. Just as in system
development, MIS staff will need to understand
expert system concepts in order to properly test
and validate after making changes. They are also
an excellent source of knowledge about system
capacity and load patterns.
8.5.3.1.2 Knowledge Maintenance
If the system has captured all the relevant knowledge in
the task domain and if that knowledge will not change in
the foreseeable future, then knowledge maintenance will
not be needed. Few problem domains, however, are totally
static. If an expert system has been developed
specifically because the task domain is changing, then
users, experts, and the MIS department will all
participate in a prolonged Production Phase that will
incorporate multiple Evaluation Stages.
8.5.3.2 Post-Implementation Evaluation j
The system should be reviewed formally to determine whether it ,is
operating correctly and efficiently from a technical standpoint, and
whether it continues to effectively address the information
management problem. This'formal Post Implementation Evaluation is
conducted only once, during the first occurrence of the Evaluation
Stage. The project manager or software developer should evaluate
user satisfaction and task performance as well as prepare a benefit
analysis during the Post-Implementation Evaluation.
8.5.3.2.1 User Satisfaction
The degree of user satisfaction will be reflected in the
Post-Implementation Report. , .
8-9
-------
OSUER Directive #9028.OOa
8.5.3.2.2 Task Performance
The system should be evaluated for task performances
against the system specifications. The Post-
Implementation Evaluation is used to perform a
comprehensive appraisal of the system. This evaluation
addresses all tasks of the system: support of functional
and data requirements, system technical performance, and
effectiveness of system management. All specialized
expert system test, review, and validation procedures used
in developing the system are now used to evaluate it.
8.5.3.2.3 Benefits Analysis
For each evaluation in this stage a decision is made
regarding the future of the system. If it is operating
correctly and effectively, the Production Stage continues.
This decision requires appraisal of advances in technology
and estimation of cost/benefit to upgrade the system. If
the system is no longer operating correctly or
effectively, and improvements would not be cost-effective,
the life cycle proceeds to the Archive Stage.
8-10
-------
OSWER Directive #9028.OOa
APPENDIX A
EVALUATING RULE-BASED EXPERT SYSTEM SHELLS
A.I INTRODUCTION ;
This appendix describes a set of criteria for evaluating
and comparing rule-based expert system development environments
(or shells). These guidelines will assist expert system
developers in making an informed, objective decision when
selecting a rule-based shell that is well-suited to their
development objectives and requirements.
Commercially available expert system shells are
proliferating; every major and minor software developer seemingly
is in the market. Prices of the various packages range from less
than $100 to almost $100,000 the majority are sold for less
than $5,000. The capabilities and quality of the shells cover
almost as broad a range, but not in direct proportion to price.
With the constant emergence of new packages and frequent upgrades
to older ones, the market is extremely volatile; buyers must
beware of poorly supported or otherwise inferior software. Since
there are no standards against which to evaluate a particular
shell and different shells are best suited to different types of
problems, a strategy must be developed to address the problem.
The evaluation of. expert system shells differs from that
of languages and other programming tools because of the lack of
standards. The challenge for the system developer is to
carefully consider the characteristics of his project and then
match those -characteristics to a shell that is appropriate. If
the type of problem to be addressed by the shell is known, then a
small benchmark rule base can be designed for implementation on
the various shells being evaluated. In this way a standard of
comparison can be loosely set. If the problem type is unknown or
varied, then the shell must be evaluated in a more abstract
fashion.
This appendix describes the major factors that should be
considered as part of a comprehensive evaluation of rule-based
expert system shells. Two attachments accompany this appendix:
the first is an evaluation form that incorporates the major
evaluation factors in a convenient checklist; and the second is a
comparison of expert system application types to shell
characteristics that is included to aid the developer in
determining what type of shell is best suited to a given problem.
The conclusion provides some general comments and recommendations
to guide expert system developers in the evaluation process.
A-l
-------
OSUER Directive #9028.OOa
A.2 SHELL EVALUATION FACTORS
This Section briefly describes the important factors that
should be considered in the comprehensive evaluation of a rule-
based expert system shell. These factors have also been listed
in Attachment A.I, Expert System Shell Evaluation Form. In
evaluating a particular shell, the evaluator should quantify
whatever values possible in order to enhance the objectivity of
the evaluation. The descriptions below attempt to summarize the
types of issues that should be addressed in each Section of the
form.
A.2.1 General Information
Attachment A.I, Sections 1 through 4 provide an outline
for recording general information. Since the shells being
considered may be reviewed by more than one person or by persons
unknown to each other, record should' be kept of the evaluators.
Vendor contacts should be listed in order to provide a source for
further purchase or evaluation information. Any impressions of
the vendor should also be noted, such as how long the shell has
been on the market or general impressions about the quality of
the vendor's product line. Hardware and software requirements
should be noted also. Attention should be paid to the minimal
and recommended configuration and the actual configuration used
to test the shell.
A.2.2 Features
A survey of the features present in the shell is an
important initial step in assessing the shell. The key features
that should be evaluated are described below. These features are
listed in Section 5 of Attachment A.I. . .
A.2.2.1 Structure (Attachment A.I, Section 5.1)
Different software tools use different types of
parameters to represent values. Many, such as logical
and string variables, are commonly supported by expert
system shells, but some common ones, such as arrays and
floating point numbers, frequently are not. In order to
determine which data types are required, the evaluator
must fully define the problem domain in advance. If that
is impossible, then all data types that may possibly be
needed should be considered.
Flexibility in the formation of rules varies widely among
shells. Some do not permit multiple clauses in the
A-2
-------
OSUER Directive #9028.00a
premise (IF) Section of the rule, or make allowance for
an alternate conclusion (ELSE, ELSE IF). Once again, the
particulars of the problem to be addressed will dictate
what rule structure is needed, but greater flexibility
will permit more general application of the package.
Many expert systems must be capable of handling uncertain
or incomplete data. The method used by a particular
shell to address uncertainty is an issue of concern. How
does the shell treat confidence factors (if at all)?
Does the shell permit probabilities only in the rule
premise? What formula does it use to combine
probabilities? What scales (e.g., -1 to +1, 0 to 100)
are available?
Some applications may require extensive mathematical
capabilities. The accuracy of mathematical calculations
and comparisons will vary depending on the internal
representation method; a variable may display a value of
OcOOO but still not act as a zero flag because its actual
value is 0.0001.
Frequently expert systems rely on external software to
perform data calls and mathematical operations. A
desirable shell feature is the ability to link to popular
applications packages such as 1-2-3-or dBASE III, or
access routines written in programming languages such as
C, Pascal, or assembler. Another consideration is
whether the shell can access the underlying operating
system and utilities such as the system clock.
The type of problem to be addressed generally determines
the type of knowledge representation to be used. There
are problems that may be addressed in several ways and
the differences between the methods of knowledge
representation are sometimes subtle. Rules, frames, and
scripts are the most widely .used representation methods,
but other obscure methods are sometimes encountered.
Rule-based shells are the most easily learned and their
knowledge bases are the simplest to understand.
Two methods of reaching conclusions are generally found
in shells: forward chaining and backward chaining.
Forward chaining shells are best suited to data driven
problems. Backward chaining shells are best suited to
diagnostic problems wherein a hypothesis is tested by
stepping backward through the rules. It is important to
note that either search method can be implemented in the
other, although not very efficiently. Some packages are
A-3
-------
OSVER Directive #9028.OOa
capable of both methods of search, either in turn or
simultaneously.
Finally, an important feature to consider is the overall
si.ze of the shell. The number of rules that it can
manipulate in one knowledge base is a quick measure of
this, but it is sometimes a false measure. For example,
a package that can work with 2000 rules sounds superior
to a package that can work with 1000 rules until the user
realizes that the first package has no alternate
conclusion or multiple premise capabilities. The second
system can thus store the same amount of knowledge in a
much smaller knowledge base. Another factor to consider
is the ability to link smaller knowledge bases together
in order to exceed the inherent limit of the shell.
A.2.2.2 Creating the Knowledge Base (Attachment A.I,
Section 5.2)
Access to the knowledge base as it is being created is
critical. Many shells have an interactive editor built
in that provides syntax and variable scope checking as
the rules are created. Some shells provide access only
through an external text processor. The ability to store
input and output in test files can tremendously simplify
testing a system, especially when the user input is long
or tedious.
Debugging features also vary widely. Error messages can
be either insightful or cryptic, depending on the
particular shell. Identification of the location of the
error (not just reporting that the error exists) and
backward and forward tracing are variably supported. A
feature seldom found but of great value is the
identification of unreachable goal states and unusable
rules.
A.2.2.3 Processing the Knowledge Base (Attachment A.I,
Section 5.3)
Once the knowledge base has been created and made
accessible to a larger audience, it will have to provide
explanations to that audience of how and why it reaches
particular conclusions. The text displayed upon request
and the points at which such requests can be made are
important issues to be considered. Users will also
probably want a "what if" or "undo" capability in order
to explore the consequences of changing decisions.
A-4
-------
OSUER Directive *9028.00a
The performance of an expert system shell can be
evaluated in several ways. A set of standard benchmarks
should be developed to test rule processing speed,
capacity, and relative development time. Typical
benchmarks include the Tower of Hanoi problem and the
animal classification game. It is difficult to compare
the performance of shells in any great detail, but these
or similar tests give a rough indication of relative
performance.
Performance can also be gauged approximately by
determining whether the shell is compiled or interpreted.
Each has its advantages and disadvantages; the particular
needs of the user will determine which is better for a
given problem. The shell's documentation may also give a
measure of performance, possibly in the form of logical
inferences per second (LIPS).
A.2.2.4 Portability (Attachment A.I, Section 5.4)
Portability of expert systems can be addressed from
several perspectives. -The value and usefulness of
knowledge bases is greatly enhanced if they are
represented in an easily comprehensible manner. If they
are stored in a generic form (e.g., an ASCII file), .then
they can be ported across wprd processing packages and
displayed in a comprehensible form. If the rule
structure is sufficiently English-like (as opposed to
complex computer -code), it can readily be interpreted by
a non-programmer expert. While the knowledge bases
produced by most shells cannot be directly ported to
other shells, the ease with which they can be converted
to other environments should be considered. In
particular, shells written in the same language may be
more compatible than shells written in different
languages.
A.2.2.5 User Interfaces (Attachment A.I, Section 5.5)
How the development and delivery systems interact with
the developer and user are significant issues. The use
of pull-down menus, the availability of on-line help, and
a transparent operating system are among the many
desirable features that facilitate interfacing with the
system. Graphic representation of the decision process,
either in static or dynamic form, can tremendously
enhance the ability of the system to present information
to the user.
A-5
-------
OSUER Directive #9028.OOa
A.2.2.6 Documentation (Attachment A.I, Section 5.6)
As with any other type of software, good documentation of
shells is critical. The quality of the tables of
contents, indices, illustrations, and examples in the
developer's and user's references will drastically affect
the ability of the developer and user to learn the shell
and solve the problems that inevitably arise. On-line or
hard copy tutorials are useful features that can
significantly reduce training time.
A.2.2.7 General. Explanations, and Additional Features.
(Attachment A.I, Sections 5.7 - 5.9)
Because of the wide variety of features offered by
different shells, no fill-in-the-blanks form can
completely describe the attributes of all shells.
Therefore, the evaluator should make general comments
about the suitability of a given shell. Attachment A.I,
Expert System Shell Evaluation Form, provides ample room
for explanations of other features or descriptions of
other problems. This is sometimes the most important
part of an evaluation; the evaluators should be liberal
in their note taking.
A.2.3 Overall Evaluation and Comments (Attachment A.I,
Sections 6 and 7)
Finally, a conclusion has to be drawn about the overall
quality of the shell and its suitability to the problem at hand.
This is ultimately a subjective opinion since the evaluators may
have certain biases. Therefore, if one evaluator is going to
review each of several packages being considered for a particular
application, the same individual should evaluate them all in
order to permit a reasonably fair comparison. If time permits
several evaluators to review each shell, then a higher degree of
objectivity can be attained.
f
A. 3 CONCLUSION
The evaluation of a rule-based shell can be divided into
a review of the overall structure; the creation, processing, and
portability of the knowledge base; user interfaces;
documentation; and other miscellaneous information. This
information should be recorded on a standard form, such as
provided in Attachment A.I, in order to facilitate comparisons of
different rule-based shells. However, the evaluator must be
careful not to get caught up in the necessity of filling in every
A-6
-------
OSUER Directive 09028.OOa
single blank on the form and thus lose sight of the larger
objective of getting a feel for the suitability of the shell to a
given problem. Ultimately the question to be answered is, "Does
the package do the job as intended?" Such fundamental questions
as "Does the package work as explained in the manual?" and "Does
the package have unexplained features?" are frequently overlooked
because they are difficult to quantify.
Many commercial expert system shells were developed to
solve one specific type of problem and have since been modified
into general problem solvers. The features added to make the
shell a general purpose tool are sometimes very obviously add-
ons; the underlying functionality is not enhanced by them. The
evaluator must seek out the underlying functionality and not be
misled by the peripheral features. Remaining objective in this
manner is essential to a proper evaluation. Attachment A.2,
Shell Characteristics vs. Application Types, correlates the shell
features discussed here to typical expert system application
types as discussed in Chapter 1.
A-7
-------
OSWER DIRECTIVE *9028.00a
Attachment A.1
EVALUATION FORM
FOR RULE-BASED EXPERT SYSTEM SHELLS
1. TESTER
Name:
Office:
Date:
Phone:
2.PRODUCT
Name:
Release/Version:
Number of Installations:
Run Time Version Price:
Price:
GSA (Y/N)?
User Groups:
3. VENDOR
Name:
Address: _
Years in Business:
Support Available:
4. REQUIREMENTS
Minimal
Hardware:
Financial Status:
Training Available:
Recommended
Hardware:
Operating System:
Systems it will run on:
System tested on:
-------
Evaluation Form for Rule-based Expert System Shells OSWER DIRECTIVE #902a.00a
Page 2
5. FEATURES
If a feature is present, place an X in the box next to it If an explanation is necessary, place
a number in the box and add an explanation in the EXPLANATIONS section. If the feature is
not present and no explanation is needed, leave the box unmarked. An asterisk indicates a core
feature.
5.1 STRUCTURE
Types of parameter values
* Logical L-l
* Numeric ''
* Enumerated ''
Floating point : I»
Complex rule structures
Premise (IF) section
Multiple ANDs....: L-1
MultipleORs n
Conclusion (THEN) section
Multiple ANDs LJ
Confidence factor (deal with uncertainty)
Bayesian probability II
MYCIN formula (CF1+CF2/100*(100-CF1))... HH
Other (specify) , 'I
Multiple rule contexts ''
Rules refer-to parameters in direct contexts I
Math capability to combine or evaluate rule contexts «
Linkage to higher level programming languages (HLPL)
Use HLPL to create values or get data LJ
Use HLPL to combine parameter values IJ
Use HLPL to perform mathematical operations IJ
-------
_.....,.._ ., OSWER DIRECTIVE #9023.003
Evaluation Form for Rule-based Expert System Shells
Page3
Variety of knowledge representation forms
* Rules [[[ L- 1
Frames [[[ « '
Examples ...................... . .............................................. I I
Other (specify) [[[ I 1
Chaining mechanism
Forward only [[[ I I
Backward only [[[ I I
* Capacity (number of rules)
Object-oriented structure ............ . ...................................... ' >
Extensible expert system shell (explain) .............. . .................. ' I
5.2 CREATING THE KNOWLEDGE BASE
Interactive rule-building with error checking .................... . ............ I I
* Test file processing
Save an interactive session and replay it .......................... . ...... L I
Run a test file [[[ I
* Debugging features
Meaningful error messages.... ................... . ........................ > I
Help in locating the source of the problem. .............................. < I
Interaction between run, debug, and edit ................................ L '
Check for unused rules ........................................ '. ............ I '
-------
Evaluation Form for Ride-based Expert System Shells OSWER DWECTWE #9028.00*
Page 4
Undo capability *'
Truth Maintenance **
Performance
Compiled ''
Interpreted '
Good response time (explain) '
5.4 PORTABILITY
Knowledge base stored in ASCII file (or other standard) '«
5.5 USER INTERFACE
* Invisible operating system .' ''
Menus ''
Commands : I'
On-line help ''
* Logical, intuitive operation «>
* Protects against user errors '. I'
* Provides meaningful error message II
Graphics display capability
Show decision structure II
Show data structure , 'I
Illustrate parameter or derived values I'
Show relations between parameters 'I
Other CH
Mouse, icons, pop-up menus, windows II
5.6 DOCUMENTATION
Easy-to-use table of contents 'I
* Well organized ''
* Well written :
-------
Evaluation Form for Rule-based Expert System Shells
Page 5
Good examples [[[ I I
Illustrated.... [[[ L 1
* Good index ........... . [[[ HU
Programmer's reference [[[ I J
-------
Evaluation Form for Ride-based Expert System Shells
Page 6
6. OVERALL EVALUATION
Does this package meet your expectations '' '
, for a product of its type? Yes No
Does this package meet all the core requirements '' ''
in the evaluation form? Yes No
Do you feel that this package should be I' ''
listed as a preferred package Yes No
How many hours did it take you to become
reasonably proficient in the package's use? hours
The following items are rated, on a scale of 1 to 10. A poor rating is indicated by 1 and an
excellent rating is indicated by 10.
Performance Documentation Recommendation
Ease of Use Ease of Learning Flexibility
7. COMMENTS
-------
Shell Characteristics vs. Application Types
r Probably Required
O Possibly Required
Q Not Likely to be Required
Diagnosis &
Classification
to
'co
"J3 S
< e
nl
Q ofl
C/J
1 c <8
'co c
CD >,
Q CO
Prediction &
Simulation
Monitoring
0
CO
1
0)
to
Intelligent
Assistant
Planning &
Scheduling
2
c
o
O
Inference Approach
Backward Chaining
Forward Chaining & Reasoning
BC, FC, & Forward Reasoning
Hypothetical Reasoning
Object Oriented
Blackboard
Induction
0
0
O
0
0
o
o
o
Q
O
O
o
a
O
O
o
o
o
Q
0
0
O
o
o
o
0
0
O
0
o
o
O
O
Q
0
O
Q
Q
O
Object Description
Frames
Frames with Inheritance
Objects
Parameter Values Pairs
Logic
Rules
Certainties
o
o
o
o
o
0
o
o
0
o
o.
o
o
o
o
o-
o
Q
O
0
o
O
0
o
o
o
o
0
o
o
o
o
o
0
o
o
o
0
Q
O
o
o
0
Q
O
O
o
O
O
0
o
o
o
Actions
Rules
Examples
Logic
Messages
Procedures
o
0
0
o
o
o
o
Q
0
o
o
o
o
0
0
o
0
o
Q
O
0
o
to
8
CD
-------
OSUER Directive #9028.OOa
APPENDIX B
GLOSSARY
AZ. See artificial intelligence.
Algorithm. A formal procedure guaranteed to produce correct or
optimal solutions.
Architecture. (1) The organizing framework imposed upon
knowledge applications and problem-solving activities. (2) The
knowledge-engineering principles that govern selection of
appropriate frameworks for specific expert systems.
Artificial intelligence. The subfield of computer science
concerned with developing intelligent computer programs. This
includes programs that can solve problems, learn from experience,
understand language, interpret visual scenes, and, in general,
behave in a way that would be considered intelligent if observed
in a human.
Backtracking. A search procedure that makes guesses at various
points during problem-solving and returning to a previous point
to make another choice- when a guess leads to an unacceptable
result. .
Backward chaining. An inference procedure or strategy .used by
the inference engine where the system starts with what it wants
to prove (e.g., the goal), and tries to establish the facts it
needs to reach that goal. The text of the goal is matched
against the conclusion (i.e., in the example "if the sky is
cloudy, then the forecast might include rain" is a conclusion)
for each rule to determine whether that rule will contribute
information to the resolution of the goal. If the conclusion of
a rule matches the goal, then the premises of the rule are
considered in turn. Each of the premises is considered to be an
intermediate goal. Results of evaluating each goal are stored in
memory to be used when evaluating subsequent goals.
Baseline. The set of life cycle products and other related
documentation which officially comprises the system at a given
point in time.
Blackboard. A database globally accessible to independent
knowledge sources and used by the: to communicate with one
another. The information they provide each other consists
primarily of intermediate results of problem solving.
B-l
-------
OSUER Directive #9028.OOa
Chaining, backward. See backward chaining.
Chaining, forward. See forward chaining.
Conflict resolution. The technique of resolving thi problem of
multiple matches in a rule-based system. When more than one
rule's antecedent matches the database, a conflict arises since
(1) every matched rule could appropriately be executed next, and
(2) only one rule can actually be executed next. A common
conflict resolution method is priority ordering, where each rule
has an assigned priority and the highest priority rule that
currently matches the database is executed next.
Control structure. Any procedure, explicit or implicit, that
determines the overall order of problem-solving activities; the
temporal organization of subprocesses.
Domain expert. A person who, through years of training and
experience, has become extremely proficient at problem solving in
a particular domain.
Domain knowledge. Knowledge about the problem domain, e.g.,
knowledge about geology in an expert system for finding mineral
deposits.
End user. The person who uses the finished expert system; the
person for whom the system was developed.
ES. See expert system.
Expert system. A computer program- that uses expert knowledge to
attain high levels of performance in a narrow problem area.
These programs typically represent knowledge symbolically,
examine and explain their reasoning processes, and address
problem areas that require years of special training and
education for humans to master.
Expertise. The set of capabilities that underlie the high
performance of human experts, including extensive domain
knowledge, heuristic rules that simplify and improve approaches
to problem-solving, metaknowledge and metacognition, and complied
forms of behavior that afford great economy in skilled
performance.
Expert system development environment. The programming language
and support package used to build the expert system.
Explanation. Motivating, justifying, or rationalizing an action
by presenting antecedent considerations such as goals, laws, or
B-2
-------
OSUER Directive #9028.OOa
heuristic rules that affected or determined the desirability of
the action.
Explanation facility. That part of an expert system that
explains how solutions were reached and justifies the steps used
to reach them.
Fact. A proposition or datum whose validity is accepted.
Forward chaining. One of several control strategies that
regulate the order in which inferences are drawn. In a rule-
based system, forward chaining begins by asserting all of the
rules whose IF clauses are true. It then checks to determine
what additional rule might be true, given the facts it has
already established. This process is repeated until the program
reaches a goal or runs out of new possibilities.
Frame. A knowledge representation method that associates
features with nodes representing concepts or objects. The
features are described in terms of attributes (called slots) and
their values. The nodes form a network connected by relations
and organized into a hierarchy. Each node's slots can be filled
with values to help describe the concept that the node
represents. The process of adding or removing values from the
slots can activate procedures (self-contained pieces of code)
attached to the slots. These procedures may then modify values
in other slots, continuing the process until the desired goal... is
achieved. Also called Data Directed Reasoning.
Heuristic. A rule of thumb or simplification that limits the
search for solutions in domains that are difficult and poorly
understood.
Inference engine. That part of a knowledge-based system or
expert system that contains the specific procedures and
algorithms for using the rules/heuristic in the knowledge base to
solve a problem. The inference engine processes the domain
knowledge (located in the knowledge base) to reach'new
conclusions.
Knowledge. The facts, beliefs, and heuristic rules a computer
program must have to behave intelligently.
Knowledge acquisition. The process of extracting, structuring,
and organizing knowledge from some source, usually human experts,
so it can be used in a program.
Knowledge base. The portion of a knowledge-based system or
expert system that contains the domain knowledge.
B-3
-------
OSWER Directive #9028.OOa
Knowledge-based system. A program in which the domain knowledge
is explicit and separate from the program's other knowledge.
Knowledge engineer. The person who designs and builds the expert
system. This person is usually a computer scientist experienced
in applied artificial intelligence methods.
*
Knowledge engineering. The discipline that addresses the task of
building expert systems; the tools and methods that support the
development of an expert system.
Knowledge management. The process of formally controlling any
changes or additions to the knowledge base in order to maintain
expert system integrity.
Knowledge representation. The process of structuring knowledge
about a problem in a way that makes the problem easier to solve.
Knowledge source. Generally, a body of domain knowledge relevant
to a specific problem. In particular, a codification made
applicable for an expert system.
Natural language. The standard method of exchanging information
between people, such as English (contrasted with artificial
languages, such as programming languages).
PROLOG. An Artificial Intelligence programming language based on
predicate calculus. PROLOG is short for the French words
Proqramination en Locricrue.
Real-world problem. A complex, practical problem which has a
solution that is useful in some cost-effective way.
Representation. The process of formulating or viewing a problem
so it will be easy to solve.
Robustness. That quality of a problem solver that permits a
gradual degradation in performance when it is pushed to the
limits of its scope of expertise or is given error laden,
inconsistent, or incomplete data or rules.
Rule. A formal way of specifying a recommendation, directive, or
strategy, expressed as IF premise THEN conclusion or IF condition
THEN action:
Rule-based methods. Programming methods using IF-THEN rules to
perform forward or backward chaining.
B-4
-------
OSUER Directive *9028.00a
Rule-based program. A computer program, that constitutes a module
of heuristic knowledge.
Search. The process of looking through the set of possible
solutions to a problem in order to find an acceptable solution.
Shell. The common term for expert system development
environment.
Symbol. A string of characters that stands for some real-world
concept.
Symbolic reasoning. Problems solving based on the application of
strategies and heuristic to manipulate symbols standing for
problem concepts.
Tools for knowledge engineering. Programming systems that
simplify the work of building expert systems. They include
languages, programs, and facilities that assist the knowledge
engineer.
Tree structure. A way of organizing information as a connected
graph where each node can branch into other nodes deeper in the
structure.
User. A person who uses an expert system, such as an end-user, a
domain expert, a knowledge engineer, a tool builder, or a
clerical staff member.
B-5
-------
OSWER DIRECTIVE #9028.003
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
Part 3: Practice Paper
Data Management
During the System Life Cycle
January, 1989
J
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
TABLE OF CONTENTS
t
1. Introduction 1
2. Selecting A Data Management Approach 5
3. An Overview Of Data Management Topics ' 15
4. Data Modeling Activities 23
5. Data Design Activities 29
6. Data Stewardship Related Activities 32
7. Data Documentation Activities 40
8. Terms And Reference Materials 44
EXHIBITS
1-1 Practice Paper Overview 2
2-1 An Overview of Data Management During the System Life Cycle 6
2-2 Scoping Your Project's Impact 8
2-3 Selecting Your Data Management Approach 9
2-4 Data Management Plan Outline 13
3-1 Data Management Topics Through the Life Cycle 16
4-1 Entity List Example 24
4-2 Conceptual Data Model Example 26
4-3 Logical Data Model Example 27
6-1 The Data Steward Role 34
6-2 Functional Data Roles 35
6-3 Determining Data Stewardship 39
7-1 Data Dictionaries in the System Life Cycle 43
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 1
INTRODUCTION
1.1 Practice Paper Purpose
This practice paper provides details to project managers concerning their
responsibilities for data management1 under the Office of Solid Waste and
Emergency Response (OSWER) System Life Cycle Management Guidance. The practice
paper describes data management during the system life cycle, and provides
guidance concerning major topics that should be addressed by project teams.
Data management begins during the Concept Phase, proceeds as requirements are
defined and software is implemented, and continues until the application system
is terminated or replaced.
This practice paper has three primary purposes:
o Focus each project manager's attention upon ensuring that the data
provided by OSWER systems meets program requirements;
o Facilitate successful data management for each project;
. o Provide a common approach to data management across OSWER.
This practice paper constitutes a section of Part 3 of the Office of Solid
Waste and Emergency Response (OSWER) System Life Cycle Management Guidance.
1.2 Practice Paper Topics
Exhibit 1-1 provides an overview of the structure of this practice paper.
Topics addressed in this practice paper include:
o Selection of an approach to data management for a project;
o The details of data management topics applicable to OSWER's Life Cycle
Management Guidance;
o The flow of major data management activities,, including data modeling,
data design, data stewardship, and data documentation.
*0ata management includes data-related activities such as logical data modeling
during requirements definition, data base design, data base management, and the
documentation of data-related decisions and products.
1 May 31, 1988
-------
EXHIBIT 1-1
DATA MANAGEMENT PRACTICE PAPER OVERVIEW
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
PRACTICE PAPER
FOR
DATA MANAGEMENT
DURING
THE SYSTEM LIFE CYCLE
Prepared for
Office of Program Management and Technology
Information Management Siajf
CIIAITER «
TEHMS
AND
REfKkKNCE
MATERIALS
CHAPTER 7
DATA DOCUMENTATION
ACTIVITIES
CHAPTER t
DATA STEWARDSHIP
RELATED ACTIVITIES
CHAPTER 5
DATA DESIGN
ACTIVITIES
CHAPTER 4
DATA MODELING
ACTIVITIES
CHAPTER 2
SELECTING
A DATA MANAGEMENT
APPROACH
CHAPTER I
INTRODUCTION
CHAPTER 3
AN OVERVIEW OF
DATA MANAGEMENT
TOPICS
-------
OSWER DIRECTIVE #90SB.OCIa
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
1.3 Data Management Responsibilities
Project managers are responsible for implementing the guidance within this
practice paper when OSWER systems and data bases are built. Each project
manager will select an appropriate approach to data management for her/his
project and ensure that this approach is documented in a Data Management Plan.
Project managers will use the Project Management Plan to schedule the activities
detailed in the Data Management Plan.
Project managers are also responsible for ensuring that a programmatic
requirement for data is identified and documented before data is collected,
processed, stored, and distributed. They must work closely with the OSWER
program offices affected by a project to ensure that this basic responsibility
is met.
OSWER program offices are responsible for supporting the data management
process of each project. This responsibility includes providing competent
programmatic personnel to identify data requirements and to define the meaning,
allowable values, edit criteria, and the- level of quality and security of the
data. Program offices that assume data stewardship responsibility (see Chapter
6) also determine who will collect data and who will ensure its integrity after
it is collected.
1.4 Basic Principles
The data management activities performed during the systems development life
cycle are based upon the following basic principles:
o Data is a valuable resource. Data is collected, stored, and used to
support critical OSWER program activities and decisions, making-accurate
and timely data an important OSWER resource.
o Data is defined separately from the technology used to collect and store
it. OSWER data requirements are recorded clearly prior to designing
automated data collection and storage methods, so that program needs are
understood and recorded. i
o Accurate information about .data is essential. Effective management of
data collected by OSWER requires that accurate information about data
(metadata) be kept.
o Common data management guidelines, methods, and tools are used. A common
approach to defining, modeling, designing, and documenting data will
improve OSWER's data quality and make it easier to share data among
systems and offices.
1.5 Why Focus Upon Data Management?
Automated and manual systems provide information to the OSWER program. This
information is used, for example, to make decisions affecting public health and
safety, environmental quality, and the use of public funds. Without this
information, the Office of Solid Waste and Emergency Response could not perfora
its mission. The data collected, created, stored, processed, and disseminate"
by OSWER systems are used to create the information OSWER needs to operate.
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Since Information must be based upon accurate data, future OSWER-funded system
development projects will focus upon the provision of accurate and timely data
to OSWER programs.
Benefits of increasing the focus upon data management include:
o Ensuring that data collected and disseminated on behalf of OSWER meets
programmatic requirements fully, including requirements for accuracy and
timeliness;
o Improving management decision-making by providing better access to more
accurate and timely data;
o Increasing productivity in the information collection and processing
activities of OSWER offices as the understanding and use of available
data increases;
o Ensuring that existing data can be shared to the maximum practicable
extent, avoiding the cost of redundant data collection and storage;
o Reducing the cost of system maintenance and the time needed to modify
implemented systems by designing more stable and flexible data bases.
1.6 How to Use this Practice Paper
Before you read this practice paper, read the OSWER System Life Cycle
Management Guidance, including the Data Management Plan exhibits within
different sections of the guidance. Then, read this practice paper to learn
more information about the topics covered during the system life cycle.
Use this practice paper to guide the development of a data management
approach that is appropriate for your project. Then, document your approach in
the project's Data Management Plan as the project evolves. Add information to
the plan, and modify existing information to reflect the current approach at any
point in time.
1.7 Project Participation and Coordination Practice Paper
Read the Project Participation and Coordination Practice Paper before
beginning to document your data management approach. The participation and'
coordination practice paper details who should be involved in project
activities, including the activities you will define in the Data Management
Plan.
1.8 Configuration Management Practice Paper
This practice paper describes change control of the baselines of
requirements, specifications, and operating functionality which are documented
in the products of the system life cycle. For instance, the Requirements Data
Dictionary is a product which is initially created during the Definition Stage.
Changes to this product are controlled using the procedures described in the
Configuration Management Practice Paper. Read the Configuration Management
Practice Paper before preparing your Data Management Plan.
-------
uJWER DIRECTIVE #9028.00a
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 2
SELECTING A DATA MANAGEMENT APPROACH
2.1 Introduction
This chapter provides guidance for the project manager who is preparing an
approach to managing data-related activities, products, and decisions during the
systems life cycle. Your data management approach will have a major impact upon
the success of your project and the information provided to OSWER programs.
2.2 What Is a Data Management Approach?
The data-related activities, products, and decisions you decide to address
during the system life cycle constitute your data management approach. Your
approach also includes the degree of r.igor you follow when performing these
activities and the level of formality you choose when documenting data-related
life cycle products and decisions.
Exhibit 2-1 provides an overview of data management activities during the
.life cycle. Chapter -3 provides detailed information about data-related
activities and products on a stage-by-stage basis.
2.3 Why Choose A Data Management Approach
Selecting and implementing a data management approach that is appropriate to
your project is'a key to your project's success. If you choose an approach that
doesn't address data dictionary issues as part of a large, high impact project,
you will increase the risk of time and cost overruns for your project, and
likely will increase maintenance costs for the completed system and its data.
This chapter will give you guidance in determining how much rigor and formalism
are required for your project.
One criterion which stands but as a major factor in determining your data
management approach is the degree of data sharing. Data sharing includes use of
one information system's data by a second system, and using the same data by
multiple functions using a single information system. If data sharing- is
extensive, you should choose a rigorous and formal data management approach.
Following this type of approach will minimize unexpected, negative impacts upon
your system and the OSWER programs it will support.
2.4 Determining Your Data Management Approach
Adjust your project's data management approach to fit the scope of the
system 'you develop. The degree of data sharing, the scope of organizational
impact, and cost of the planned system should be reviewed to determine your data
management approach. As the relative degree of each criterion increases, the
level of rigor and formalism you select for your data management approach should
.increase accordingly.
May 31, 1988
-------
EXHIBIT 2-1
AN OVERVIEW OF DATA MANAGEMENT ACTIVITIES
DURING THE SYSTEM LIFE CYCLE
INITIATION
Identify
Information
Need
Define
High Level
Data
Requirements
Archive
Data&
Son ware
EVALUATION
DEFINITION
Evaluate
Data Base
Define
Detailed
Data
Requirements
DATA
MANAGEMENT
DURING THE
SYSTEM LIFE CYCLE
PRODUCTION
Operate
and Maintain
Data Base
Design
Physical
Data Base
Create
and Test
Data Base
Convert
Production
Data
Establish
and Test
Production
Data Base
DEVELOPMENT
IMPLEMENTATION
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
2.4.1 Scoping The Project
Begin by determining the scope of data sharing, the organizational
impact, and the cost of the project. Extensive data sharing often increases the
impact of a system upon the agency. Cost is not as important a criterion in
scoping your project, but a large investment in a system should be accompanied
by a formal approach to planning and designing the data used to support the
system.
Refer to Exhibit 2-2, a chart that will help you scope your project.
If any of the statements listed under the high impact column (column"two) are
true for the project, then your project should be considered high impact.
If your project is not high impact, evaluate each criterion in column
one against the information in the medium impact column (column three). If any
of the statements under column three apply to your project, then it is a medium
impact system. If your project is neither high or medium impact, it is a low
impact system for the purpose of determining a data management approach.
2.4.2 Select A Preliminary Data Management Approach
Next, select the data management approach (preliminary) you will use
for the project. Refer to Exhibit 2-3 to help select a preliminary data
management approach. Choose the decisions, products, and activities you should
include in your data management approach, and determine'the degree of formalism
you will use to document decisions and products. Keep in mind that a primary
objective of every system project is to provide accurate and consistent data to
the system users.
High Impact Projects
Follow the guidance provided in Chapters 3 through 6 of this practice
paper if you have a high impact project. Conceptual data modeling, logical data
modeling, and physical data base design are performed. Document the plans,
activities, decisions, and products called for in these chapters formally. Data
documentation is stored in automated requirements, design, and production data
dictionaries. If you are managing a high impact project, appoint a full-time
person to coordinate data management activities for the project on your behalf.
Medium Impact Projects
Medium impact projects require subjective judgments concerning the
components you select for your data management approach. While conceptual data
modeling, logical data modeling, and physical data base design should all be
performed, the level of formalism you select for documenting your decisions and
products may be less. If you are managing a medium impact project, you may
complete data dictionary documentation differently than for a high impact
project. Although requirements, design, and production data dictionaries should
be kept, only the design and production dictionaries need to be automated.
-------
EXHIBIT 2-2
SCOPING YOUR PROJECT'S IMPACT
CRITERION
HIGH IMPACT
MEDIUM IMPACT
LOW IMPACT
Data Sharing
Dau are created, collected, and/or
used to support operations of
multiple OSWER program offices,
legions! offices, and/or state agencies
and
Data are used by multiple EPA
offices to provide environmental
information to OSWER management,
the public, and Congress
Data are created, collected, and/or
used by multiple OSWER program
offices. Regional offices, and/or
state agencies
>ata are created, collected
and/or used within a single
OSWER program office;
o dau are created by or
ollected from Regional
iffices, and/or state
igencies
Organizational Impact
Deemed a national priority system.1
and is placed on the President's list
of national priority systems
Requires hit support from
multiple OSWER program
offices, or from Regional
offices, and/or from state agencies
Requires FTE support from
a single OSWER program
office; no FTE support
required from other
OSWER offices. Regional
offices,-or from state
agencies
Cost
Cost exceeds the available budget
support of the sponsoring
OSWER office, or the total life
cycle cost meets OMB reporting
requirements
Sponsoring OSWER office
can provide sufficient
funding with available
budget support, and all
costs are below OMB
reporting requirements
1 A national priority system is a system deemed to be critical to the government's mission.
Please note: If any criterion under a column is met, then the project's application is treated as the type of project described in that column.
8
-------
EXHIBIT 2-3
SELECTING YOUR DATA MANAGEMENT APPROACH
PROJECT SCOPE
DATA
MANAGEMENT
ACTIVITIES
DOCUMENTATION
OF
DECISIONS/PRODUCTS
DATA
DICTIONARY
DOCUMENTATION
HIGH IMPACT
RIGOROUS
EXECUTION
(See Chapters 3, 4,5,6)
MEDIUM IMPACT
MODIFIED
LOW IMPACT
MODIFIED
FORMAL
(See Chapter 3)
MODIFIED
MODIFIED
FORMAL -
(See Chapters 3,7)
Requirement, Design, and
Production data dictionary
in automated form.
FORMAL
Requirement, Design, and
Production data dictionary
in automated form.
FORMAL
Production data dictionary
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCtE
Low Impact Projects
Low impact system projects require a less rigorous approach to
activities than the other two classes of projects. Logical data modeling should
be done with the resulting data requirements documented. Documentation of
production data bases should also be produced and stored manually or in an
automated data dictionary. Ensure that the activities and products you select
for your approach are appropriate to the scope of your project.
2.4.3 Consult With The OSWER Data Administrator
After selecting a preliminary data management approach, contact the
OSWER data administrator in the Office of Policy, Management and Technology for
assistance. Review the project's scope and your preliminary data management
approach with the data administrator. The data administrator will help you
adjust your data management approach to fit the scope of your project and advise
you of support capabilities OSWER can provide for your project.
2.4.4 Finalize and Document Data Management Approach
Finalize your data management approach and document it in the project's
data management plan. Exhibit 2-4 .at the end of this chapter provides the
outline of topics for your data management plan. This plan will provide
valuable input into project planning and staffing activities, including the
preparation of the Project Management Plan.
2.5 Prepare the Data Management Plan
OSWER intends to increase the degree of data integration and sharing to
provide improved information management for mission support. Implementation of
increased data integration and sharing will be supported by automated software
tools for system and data base development, and an OSWER-wide inventory or
directory of the data OSWER has collected. Plan to take advantage of these
capabilities when you prepare your data management plan.
Record the data management approach in your data management plan early in
the Concept Phase. Include those topics described in Chapter 3 which are
appropriate. Then, refer to the plan at the beginning of each stage, revising
it as you find necessary. See Exhibit 3-1 for a summary of this creation and
revision process.
As a project manager, you will find the details in the data management plan
helpful when you select staff for your project. Before you prepare your plan,
you should read Chapters 4 through 8 of this practice paper to understand the
work you .will be planning.
Pay particular attention to coordinating activities in four major areas when
preparing the substance of your data management plan.
2.5.1 Data Stewardship
Before you can be sure you have identified the relevant data requirements
for a project, you must first determine who is authorized to define these
requirements. Data stewardship assigns the functional data roles andj
10
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
responsibilities to the organizations which exercise control over data on behalf
of OSWER.
Identify the data steward, definer, and primary user for each high-level
data requirement identified during zhe Concept Phase. Consult with the OSWER
data administrator and read Chapter 6 before planning activities related to data
stewardship.
2.5.2 Data Modeling
Data modeling activities include identifying data requirements, defining
the meaning of the requirements, and structuring the data requirements into
logical models. These logical models represent the mission-oriented structure
of the data. Data modeling and data models are completely independent of all
technology used to store and access data.
When planning data modeling activities, assign an individual with systems
analysis skills and OSWER program knowledge to lead the effort. Be sure that
"users" of the system and data assume the data definer role described in Chapter
6. It is the user perspective which is critical to the success of data
modeling.
You begin data modeling in the Concept Phase and finish the first version
of the data model in the Definition Stage. The data model is modified at any
point in the life cycle when data' requirements change. Since a number of
requirements changes will normally occur during the Design and Development
stages, you should also anticipate changes to your logical data model during
these stages. See Chapter 4 ,of this document for more information on data
modeling.
2.5.3 Physical Data Base Design
Physical data base activities build upon the results of the data modeling
activities to produce a physical data base design. This data base design should
be done before creating data definition language statements that will be used to
create the physical structure of data bases. Data design activities should not
begin until after the first version of the data model is complete, since data
design activities build upon the results of the data model.
/
The individual assigned responsibility for data base design must have
technical expertise in the data base management system or file structures being
used. You should schedule walkthroughs of the design, just as you schedule
walkthroughs of your software program designs. The physical data base design
walkthroughs should be attended by the data designer, programmers who will use
the data structures, the lead data modeling person, and the person responsible
for data management.
An initial version of the data base design will normally be required by
the end of Design Stage, so that physical data structures (data bases) can be
created to support programming and unit testing during Development. See Chapter
5 for more information on data base design activities.
11
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM'LIFE CYCLE
2.5.4 Data Documentation
Collecting, storing, and using data about data (metadata) 1s crucial to
the success of system development project activities. As the Individual with
primary data management responsibility for your project, you should plan for and
manage metadata effectively. Define your data documentation activities early in
the Concept Phase, and record them in the Data Management Plan.
You should record metadata describing the meaning, attributes, properties
and structures of the data in the logical data model and data base design, as
well as data stewardship information. Normally, metadata will be stored in
automated software systems called data dictionaries. You must plan to use the
metadata of prior stages in subsequent stages whether automated software 1s used
or not. See Chapter 7 for more information about data documentation activities.
12
-------
EXHIBIT 2-4
DATA MANAGEMENT PLAN OUTLINE
This topical outline contains all the topics that Tiay be contained in the
Data Management Plan for a High Impact Project. Taken together, the topics you
enter into your Data Management Plan document your project's data management
approach.
o Information Need
Document the Information Need
Missions Supported
Scope of the Need
o Data Steward Organizations
Lead Organization Responsibilities
Other Organizations Roles
Data Definers For The Project
o Concept Phase
Entity List
Entity Definitions
Entity Identifiers
Conceptual Data Model '
Likely Sources of Data
Information Flow/Data Model Validation
Data Distribution Plan
Information Collection Burden .
o Definition Stage . '
Interview Plans
Data Analysis By Process
Entity Normalization
Conceptual Data Model Revision
High-Level Data Requirements Revision
Logical Data Model
Requirements Data Dictionary
Data Flow/Logical Data Model Validation
o Design Stage
Logical Data Model Revision
Physical Data Base Design
Design Data Dictionary
o Development Stage
Data Structures for Programming Support
Data (structure) Revision Approach
Data Backup, Logging and Recovery Plans
o Implementation Stage
Testing Support (See Testing Support Plan)
Cutover Plans
Production Data Dictionary
13
-------
EXHIBIT 2-4
DATA MANAGEMENT PLAN OUTLINE (Continued)
o Production Stage
Data Base and Metadata Management
Support of Configuration Management
Backup, Recovery and Restart
Role of the Custodian
o Evaluation Stage
Audit and Evaluation Support Plan
Response to Evaluation Report
o Archive Stage
Data Base DDL and Metadata Disposition
Data Disposition
Cutover Procedures
o Data Documentation Responsibilities
Creating Data Documentation
Maintaining Existing Data Documentation
o Data Quality Assurance Plans
Responsible Organization
Milestones and Staffing
Data Quality Objective Monitoring Plan
o Data Security Requirements and Strategy
Sensitive Data
o Data Life Cycle Methodologies and Tools
Metadata Management Approach
Development & Installation Phase
Data Management Software
Operations Phase
o Data Conversion Strategy
o Data Conversion Plan
Sources
Media
Load Programs Required
Schedule and Staffing
Validation
o Plan For Physical Flow Of Data
o Data Testing S.rategy
o Testing Support
Kinds of Test Data Bases Required
Test Data Provision
Performance Validation Plan
Responsible Organization
Projected Testing Support Needed
14
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 3
AN OVERVIEW OF DATA MANAGEMENT TOPICS
3.0 Introduction
This chapter provides additional details of the topics that you can select
as a part of your data management approach. If you select one of these topics,
refer to this chapter to learn what should be included in your Data Management
Plan. Keep in mind that the data management approach and much of the Data
Management Plan are prepared during the early part of Concept Phase. In many
instances you may not want to enter all of the details of your data management
approach in the plan; if not, at least indicate which topics you are addressing.
3.1 Initiation Phase Topic
3.1.1 Information Need
Prepare a brief.summary of the information need created by the information
management problem discussed in the Initiation Decision Paper. If it is
necessary to collect new information or access existing information to solve a
problem, then you should select a data management approach and document it in a
Data Management Plan. Record the programmatic missions with the information
need, and review this information at the start of each phase/stage to be sure it
is still valid.
3.2 Concept Phase Top-ics
3.2.1 Data Steward Organizations
;
Document the data stewardship responsibilities of the organization
sponsoring the project. Then, record the data steward, data collector, data
definer, and primary data user for each data entity identified during the
Concept and Definition. See Chapter 6 for more information 'about data
stewardship.
3.2.2 High Level Data Requirements
-- Entity List
- Data Entities about which data is needed (e.g., Employee)
-- Entity Definitions
- Programmatic definitions of each entity
2
Data entity is a person, place, thing, concept, or event about which OSWER will
store data.
15 May 31, 1988
-------
EXHIBIT 3-1: DATA MANAGEMENT TOPICS
THROUGH THE SYSTEM LIFE CYCLE
^PHASE/STAGE
^S.
DATA ^v
MANAGEMENT^.
PLAN TOPIC XVV
Information Need Summary
Dau Steward Organizalions
(Plan)
High Level Dau Requirements
- Conceptual Modeling
Dau Documentation Plans
Responsibilities
Life Cycle Methodology/Tool Plans
Physical Dau Flow Plan
Definition Stage
Logical Modeling
- Detailed Requirements
Dau Quality Assurance Plans
- Monitor Dau Quality
Data Security Requirements
Strategy
Design Stage
- Physical DB Design
Dau Conversion Strategy
Testing Support Plan
Dau Testing Strategy
Development Stage
Programming Support
Implernenutkxi.Slage
Testing Support
Production Cutover Plans
Production Stage
DB Support Plan
- Production Dictionary Support Plan
- Backup, Recovery, Restart Plans
(Procedures)
- Custodial Responsibilities
Evaluation Stage
- Support Evaluation
Response
Archive Stage
- Tata Base DDL. Metadata, and
Dau Disposition Plan
|
i
2
C
'
fc
1
8
c
c
c
c
c
c
1
1
a
R
R
R
R
R
R
C
C
z
S
o
R
R
R
R
R
R
C
C
C
c
i
1
§
a
R
R
R
R
R
R
R
R
R
R
C
.
5
P
I
i
-
R
R
R
'R
R
R
R
R
R
C
z
I
§
£
R
R
R
R
R
C
C
z
|
1
£
R
1
5
*
R
I.ERFNP;
C = CREATE R = RERNE/REVISE
16
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Entity Identifiers (Candidate Keys)
- Data elements used to uniquely identify each entity (e.g., Employee
SSN)
~ Conceptual Data Model
- Develop a graphic representation of entities and their relationships
Likely Sources of Data (by Entity)
- Identify the likely functions and organizations providing data
Validation of the Conceptual Data Model
- Document your plan to validate that each entity in the high-level
information flow is in the conceptual data model. The high-level
information flow is done early in the-jConcept Phase, and is included in
the System Concept Document.
Data Distribution Plan (by Entity)
- Record the use of each entity in two 2-dimensional matrices, by
function and by distribution/location.
-- Information Collection Burden
- P.Ian for any additional information collection burden hours imposed
by the solution to the information need.
3.2.3 Data Documentation Responsibilities
Produce and document plans to allocate responsibilities for documenting data
requirements, data designs and physical data structures. Look for plans done by
earlier projects of the same scope, since this can save you time and effort.
Include the- data objects to be documented, the attributes to be documented and
the functional areas responsible for creating the documentation. Also detail
the functional support available to help people create and record metadata, and
the media to be used for recording dictionary documentation. Prepare
maintenance procedures for dictionary documentation, in coordination with the
project's configuration management plan.
3.2.4 Life Cycle Methodologies and Tools
Record the methodologies and automated tools you will use during the project
to support data management activities. Be sure to include your explicit plan
for managing the flow of metadata (information about data) between methods and
tools through the life cycle. These tools might include automated data
dictionaries, data modeling and data design tools, and data base management
systems.
3.2.5 Plan For Physical Flow Of Data
Record your plans for the physical data (data sets) flowing through the
system. This is particularly important when automated data is added to the
system from external sources, or when multiple physical data sets need to be
updated from a single source of data. Later, timeliness and error correction
procedures should be developed.
17
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
3.3 Definition Stage Topics
Definition Stage activities and products are especially important, since
they define a system's detailed data requirements. Chapters 4 and 7 contain
more information about these activities. Topics covered should Include:
Interview Plans
- Determine and record the name of the interviewees, functional areas
to be the subject of interviews, and relationship between the data
- requirements interview and the functional requirements interview. (You
may want to combine these interviews.) Also record the information
needed from the interviews in a pre-written interview guide. After the
interview, follow-up with a written summary of the results to the
interviewee.
Data Analysis By Process
- Determine and document your plan to identify and analyze the data
elements required by each process during the functional analysis (see
SLCM guidance). This analysis plus the logical data model will produce
the detailed data requirements for the project.
Entity Normalization
- Determine and record the methodology you will use to normalize the
entities required by each process. Normalization is the process of
reducing a data entity to its most basic form, removing repeating data
elements, data elements not dependent upon the key of the entity, and
data elements dependent upon the key of other entities. Your plan to
perform data analysis should be documented as part of the data analysis
topic mentioned above.
Conceptual Data Model Revision
- Determine and document your plan for updating the conceptual data
model as new entities are determined. The revision should be on-going
during the Definition Stage, not held until the end of the stage.
Logical Data Model
- Determine and document details of your plan to create logical data
models for each functional process, and for the project as a whole. A
logical data model is a graphic depiction of the logical, or
programmatic, data needed to support an organizational mission. See
Chapter 4 for more information. Once you have a draft of your logical
data model, hold group review sessions (walkthroughs) to validate it.
Requirements Data Dictionary
- Determine and document your plan for recording metadata describing
data entities and data elements in the requirements data dictionary. Be
specific about responsibilities, activities, and external support you-
will need.
Data Flow/Logical Data Model Validation
- Document your plan to validate the data elements in the logical data
model against the data elements in data flows prepared by the system
analysts.
18
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
3.3.1 Data Quality Assurance
Much of the improvement in data quality that OSWER requires will occur as a
result of following this practice paper's approach carefully. However if your
system users want to measure the quality of the data in their system
periodically, document your data quality assurance plan. If you establish
quantitative measures of quality for specific data elements, or combinations of
data elements, record your plan for measuring your data bases' ability to meet
these measures. Include the following information:
Responsibilities
- What organization is responsible for monitoring data quality
- Who will monitor or audit data quality
Data Quality Objective Monitoring Plan
- Objectives of the data quality assurance plan
- What data will be monitored
- How will it be done and how often
- How will you resolve problems that are raised
- Who will monitor problem resolution
3.3.2 Data Security Requirements and Strategy
include your plan for identifying data security requirements, recording the
requirements, and implementing them. Responsibility for identifying sensitive
data and protecting the data must be detailed in accordance with the data
stewardship roles of the project. Details of data security requirements are
recorded in the Security Manual during the Development Stage.
3.4 Design Stage Topics ,
Create and document your approach to Design Stage data management topics.
Topics should" include:
i
3.4.1 Logical Data Model Revision
- Detail your plan for revising the overall logical data model.
3.4.2 Physical Data Base Design (Data Design)
- Document the method you plan to use to prepare a physical data base
design (data design) for the project. The physical data base design
you prepare will be used to create the physical data structures to
support your system. See Chapter 6 for more information on this topic.
3.4.3 Design Data Dictionary
- Enter - your plan for recording information (metadata) about the
physical data base design in the design data dictionary. Include an
outline of what metadata can be copied from the requirements data
dictionary.
3.4.4 Role of the Data Custodian
- Identify and document the data custodian for data in the data base.
With distributed data bases you will have multiple custodians, so
careful planning is needed. Record this information in the design data
dictionary during the Design Stage.
19
-------
OSWBfDIRECTIVE #3023.005
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
3.4.5 Data Conversion Strategy and Plan
If existing data will be used in the new system, include details of
your plans for data conversion and the support of data conversion
activities. Information to be included are:
Sources
~ Media
Load Programs Required (automated procedures)
-Edit Criteria Related to Quality
Validation Plan
3.4.6 Data Testing Strategy
Document your general strategy for testing data base performance and
functionality to prove you have a viable data base design that meets detailed
data and system functional requirements. Most functional test planning
information will be stored in the System Test Document.
3.4.7 Testing Support Plan
Record your detailed plans for supporting testing from the Development Stage
through the Production Stage. Information includes:
Kinds of Physical Test Data Bases Required
Test Data Provision
Performance Validation Plan
Projected Testing Support Needed
3.5 Development Stage Topics
Your approach to Development Stage data management activities should
include:
3.5.1 Data Structures for Programming Support
- Record your plan to support software program development and unit
testing here. Coordinate your plan carefully to avoid problems. This
section will vary depending upon the data base and programming language
you use.
*
3.5.2 Data (structure) Revision Approach
- Document the details of the change control procedures you will use to
support changing and adding data structures to the data base during
this stage. Refer to the project's Configuration Management Plan for
guidance concerning the overall framework you will use. Also document
your approach to updating the design data dictionary to keep abreast of
all changes.
3.5.3 Data Backup, Logging and Recovery Plans
- Record your plans for backup, logging, and recovery of physical data
sets stored in the data structures your project creates. It is
suggested that you document your plans for test data bases, and your
plans for the production versions of data.
20
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
3.5.4 Data Documentation and User Training Materials
Ensure that data management concerns are included in all documentation,
not just the data dictionaries, and that user training material
includes an emphasis on data creation, collection, validation, and
quality assurance issues.
3.6 Implementation Stage Topics
Your approach to Implementation Stage data management topics covered in your
plan should include:
3.6.1 Testing (See Acceptance Test Plans)
- Include plans for supporting integration testing and acceptance
testing with the data bases that have been built.
3.6.2 Cutover Plans
- Record the activities the data base administration function must
carry out to support cutover to the production system. This cutover
will usually involve unloading test data, securing the production
version of software, and loading 'data that has been converted from an
existing system.
3.7 Production Stage Topics
Topics covered should include:
3.7.1 Data Base and Metadata Management
- Be sure to have procedures, and.plans for supporting the production
data base and keeping the production data dictionary accurate. Also
describe procedures to keep the design and production data dictionaries
in synchronization when enhancements occur.
3.7.2 Support of Configuration Management
- Coordinate data management with the configuration plans of the
project. Refer to the project's Configuration Management Plan for
details of the project's approach. Pay particular attention to the
updating of data dictionary information, and control of changes
requested for data bases.
3.8 Evaluation Stage Topics
Topics covered should include:
3.8.1 Audit and Evaluation Support Plan
- If data management activities are required to support the audit and
evaluation process, document the planned activities.
3.8.2 Response to Evaluation Report
- If data management related actions are required in response to points
of an evaluation report, document the planned actions.
3.9 Archive Stage Topics
Topics covered should include:
21
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
3.9.1 Data Base Data Definition Language and Metadata Disposition
- Document plans to archive or pass to another data base the data
definition language statements that create the data structures in the
data base, and the metadata in the production data dictionary.
3.9.2 Data Disposition
- Document plans to a-chive or pass to another data base the data sets
(data) in the data base.
3.9.3 " Cutover Procedures
- Document plans for the data base administration function to support
the cutover from production to archiving or another system.
22
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Chapter 4
DATA MODELING ACTIVITIES
4.1 Introduction
This chapter presents the key concepts of data modeling and lists topics
outlined 1n the Data Management Plan that pertain to data modeling.
4.2 Overview
Data modeling activities relate the data requirements of a project to a
programmatic, or end-user perspective. These activities provide essential input
into the statement of requirements for a project. There are two levels of data
modeling that you will perform during a project:
o Conceptual Data Modeling
- a broad look at data requirements
o Logical Data Modeling
- expansion of the conceptual model to include detailed requirements
You perform conceptual data modeling during the Concept Phase, and logical
data modeling during the Definition Stage. In both cases, the data models are
validated by checking their completeness against data flow diagrams prepared by
systems analysts, and by reviewing them with end-users. Conceptual data
modeling contributes to the preparation of high-level data requirements for the
System Concept, and the logical data modeling contributes to the preparation of
detailed data requirements that are documented in the requirements data
dictionary.
4.3 Conceptual Data Modeling
Conceptual data modeling, the first step in data modeling, begins during
the Concept Phase. A conceptual data model shows the kinds of data that should
be stored for system users, and the relationships among the data. Often a
conceptual data model is used to coordinate the data requirements of one project
with the requirements of other projects by providing a framework for assigning
data elements to specific, high-level data entities.
The data identified during conceptual data modeling are called data
entities. A data entity is a person, place, thing, concept or event about which
OSWER needs to obtain data. Facility, Determination, and Permit are all
examples of data entities.
You begin conceptual data modeling by preparing a list of data entities for
the project. The 11st can be prepared by examining documents describing the
processes to be automated, by interviewing programmatic staff directly, or by
working, in conjunction with systems analysts during their initial interviews of
users. Exhibit 4-1 is an example of a data entity list.
23 May 31, 1988
-------
EXHIBIT 4-1: ENTITY LIST
EXAMPLE
Award
Employee
Institution
Organization
Proposal
«W-
Part of high level data requirements,
24
-------
PRACTICE PAPER FOR DATA MANAGEMEHT DURING THE
,.*».. .
Then, each data entity is defined in programmatic terms, and the data
s that can be used to uniquely identify each occurrence of each entity
identified. Record the data entity definitions carefully, since they are
needed for the System Concept document, and for the requirements data dictionary
(Definition Stage).
Next preoare a picture of the entities, the relationships between entities,
and the data elements that uniquely identify each entity (sometimes called
candidate keys). Exhibit 4-2 is an example of such a picture. These pictures
are sometimes called entity-relationship diagrams.
Finally, review the information flows, or high-level data flow diagrams
prepared by systems analysts to ensure that data entities representing all the
data in the data flows and stores on these flow diagrams are represented in the
conceptual data model. Additional data entities are defined, added, and the
entity-relationship diagram is updated until it is complete.
4.4 Logical Data Modeling
Logical data modeling is a vital activity during the development of any
data base, since it includes analysis of programmatic data requirements and
provides the starting point for physical data base design. Modeling includes
depicting the data required for programmatic functions graphically, and
validating the accuracy . of the requirement with the users of the data. The
logical data model provides a clear, accurate description of data requirements
that physical data base designers (see Chapter 5) use to begin design of data
bases.
Logical data modeling is performed during the Definition Stage, and the
logical model is maintained throughout the life cycle. Logical data modeling
builds upon the conceptual data model diagrams and entity definitions that were
prepared during the Concept Phase. This model .extends the conceptual data model
by identifying the data elements required to describe each data entity. While
the conceptual data model might contain the data entity "Employee" and the data
element "Employee Number" that uniquely identifies the entity, the logical data
model would include other data elements, often called attributes, needed to
describe "Employee". Examples of such data elements include "Employee Name",
"Employee Home Address", and "Employee Birth Date".
Perform a detailed data analysis of the data flows and stores documented by
systems analysts during the Definition Stage to expand the conceptual data model
into a more detailed, logical data model by adding and defining the data
elements required for each entity.
Construct a logical data model diagram that includes data entities and data
elements needed by all processes. Then, "normalize" data entities to eliminate
logical redundancy, so that your "normalized" logical data model will contain
each data element in only one place. See Exhibit 4-3 for an example of a
logical data model.
When you normalize, the number of data entities will increase dramatically
as you are forced to identify more specific entities that portray sub-types
("Manager", a sub-type of "Employee") and roles ("Assigned Employee") of the
entities you had in your conceptual model. Don't be alarmed by this
25
-------
EXHIBIT 4-2: CONCEPTUAL DATA MODEL
EXAMPLE
Entity
Unique Identifier
(Candidate Key)
INSTITUTION
(External)
Authorizes
Relationship
PROPOSAL NUMBER
PROPOSAL
This conceptual data model describes the relationship between data entities composing the high level
data requirements for a proposal award function. Of particular note is the fact that one proposal can
result in many awards. Other relationships in the model are:
One Institution
One Employee
, One Organization
One Proposal is
One Institution
Submits
Reviews
Authorizes
Funded by
Receives
Employs
Many
Many
Many
Many
Many
Many
-Proposals
Proposals
Awards
Awards
Awards
Employees
LEGEND
Indicates One Entity
Indicates Many Entities
26
-------
EXHIBIT 4-3: LOGICAL DATA MODEL
EXAMPLE
PROCESS: PRINT A SUMMARY OF PROPOSALS
FUNDED LAST MONTH BY SUBMITTING INSTITUTION
RELATIONSHIP ENTITY NAME DATA ELEMENTS (ATTRIBUTES)
ro
ww
£ FKUFUSAL
g£ PROPOSAL REVI
^£ KMHiWKK
+
f ORfiANI7ATION
k.b.
£^ AWAKU
fe.
ERQJ, PRO NAME, PRO SUBMIT DATE, IINS IDI
P'RO * +- P SSN # + RRV #, REV DETERMINATION,
REV START DATE, REV END DATE
RMP SSN #f BMP NAME. EMP PHONE. JORG NAME]
QRfiJD, ORG NAME
AWDff, AWD AMT, AWD DATE, lORG ID I, iPROfll JlNS Id
INS ID. INS NAME
This is a logical data model for the process of printing a summary by submitting institution of funded proposals. The entities
here are drawn from a conceptual data model that covers many processes. Once this logical model is normalized, any new
entities found are added to the conceptual data model, and used to update the overall logical data model.
+ Indicates Cuiu .deflation
LEGEND
Entities in bold , Separates Data Elements
Primary Kcvs Underlined (Secondary Kcvs in Boxed
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
proliferation of data entities, it is a sign of your success at "normalizing"
entities. This proliferation of entities is caused by your being increasingly
more specific in describing and modeling your project's data requirements.
Validate that you have identified all data entities and elements by
examining the systems analyst's data flow diagrams again to ensure that all data
elements in the data flows and data stores have been accounted for in your
logical data model. End-users should then review the model through a formal
process, and "sign-off" on the logical model's accuracy.
Finally, add all the descriptions of the data entities and data elements in
the logical model, as well as their definitions, to the requirements data
dictionary. Contact the Information Management Staff of OPMT to obtain a copy
of relevant OSWER documentation standards. Enter data stewardship information
into the dictionary (see Chapter 6). The requirements data dictionary an'd the
logical data model diagrams taken together comprise the "detailed data
requirements for the project.
4.5 Data Management Plan Topics Related to.Data Modeling
Topics in the Data Management Plan that cover data modeling are included
below. Methodologies and tools should be selected before data modeling begins.
o Concept Development
~ Entity List
Entity Definitions .
Entity Identifiers . '
Conceptual Data Model
Likely Sources of Data
Information Flow/Data Model Validation
Data Distribution Plan
Information Collection Burden
o Definition Stage
Interview Plans
Data Analysis By Process
Entity Normalization
Conceptual Data Model Revision *
High-Level Data Requirements Revision
Logical Data Model
Data Flow/Logical Data Model Validation
o Design Stage
Logical Data Model Revision
o Data Security Requirements and Strategy
Sensitive Data
(Identified During Data Analysis and Modeling)
o Life Cycle Data Management Methodologies and Tools
28
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Chapter 5
PHYSICAL DATA BASE DESIGN ACTIVITIES
5.1 Introduction
This chapter presents an overview of physical data base design activities
and lists the topics in the Data Management Plan that pertain to physical data
base design. Physical data base design activities support the storage of data
needed by a system's users. Contact the National Data Processing Division
(NDPD) for a copy of the procedures for implementing data bases on NDPD
equipment.
5.2 Overview
Data design activities have been a part of system development practices for
some years now. The product of these activities has been data structures, for
example physical record types, that support a system's software programs. The
development of physical data base structures without data modeling has resulted
in unstable designs that increase maintenance costs, and limit the usefulness of
data stored in these structures. So don't begin your data management activities
by doing a physical data base design.
Physical data base design activities begin during the Design Stage after
the logical data model is completed. Information contained in the logical data
model and the requirements data dictionary is used as a starting point for the
design process. The designer transforms the logical data model into an initial
physical data base design that can be implemented in a data base management
system. Data definition language statements are produced during the Development
Stage to provide common data structures for programming and unit testing
support. During the Development and Implementation Stages, the design is
altered to meet the performance requirements of the system, and the data
definition language statements are also revised.
5.3 The Physical Data Base Design Process
The physical data base design process must include participation by someone
with an expert knowledge of the DBMS being used. You may decide to do physical
data base design in a single step, but if you have a large and complex system,
you may want to plan to follow the steps outlined below. There are three
significant products in the physical data base design process that you may
produce to ensure thoroughness:
o Data driven physical model
o Process driven physical model
o Location driven physical model
These three products are the focal point of the process outlined below. Please
note that in some circumstances you may combine two or three of the models and
produce a single model instead.
29 '. May 31, 1988
-------
OSWtH
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
5.3.1 Step 1: Data Driven Physical Model
Begin your physical data base design process by taking your logical data
model and transforming it to fit the structure of the data base management
system (DBMS) you are using. If you are using a DBMS that can implement a
relational data model directly, then this will not require much work at all,
since your normalized logical data model will be structured in relational terms.
With a relational DBMS the logical model is the data driven physical model. If
you use a relational DBMS and have no requirements for high-volume processing or
distributed processing, your physical data base design 1s completed.
If you are using a DBMS such as FOCUS or SYSTEM 2000 that does not
support the relational data model, you will have to transform your logical data
model into the hierarchical or network structures which these other DBMS'
support. This activity involves allocating data elements from your logical data
model to more than one physical record or segment, depending upon the DBMS being
used. Assign an individual with expert-level knowledge of the DBMS package
being used to perform this task. Ask the DBMS vendor if they can provide design
examples or material for you.
5.3.2 Step 2: Process Driven Physical Model
Review the transaction volumes and service requirements of particular
software programs to determine if high volume or very fast response time is
required for some transactions. If so, alter your data driven physical model to
place the data elements needed by the high volume or high service requirement
programs together in one physical record. This introduces phys-ical redundancy
that may be justified by .your application. Schedule a walkthrough of the
process driven physical model with systems analysts and programmers, the data
base designer, and the project's data administration staff.
5.3.3 Step 3: Location Driven Physical Model
If you plan to distribute your data base(s) at more than one location and
run software against the distributed data, then you will have to modify your
process driven physical model to account for the replication of data needed to
support distributed processing. This 1s not a trivial activity. Make certain
that the design team has expertise in the design of distributed data bases using
the selected DBMS package.
4
5.4 Using Your Physical Data Base Designs
Normally, the process driven physical model will be the initial physical
data base design for your project. If you are using a distributed data base for
your project, you will have to complete a location driven physical model and use
it for your physical data base design. Whichever is the case, use the initial
physical data base design to support the preparation of data definition language
statements for your DBMS during Development. If no DBMS is used, these
statements will be replaced by copybooks or copy libraries that can be shared by
programmers.
Once the physical data base design is done, create the design data
dictionary (if you haven't already done so), by copying the definitions for data
elements in the requirements "data dictionary into the design data dictionary and
30
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
adding technical information that describes your physical data base design.
Maintain the design data dictionary and transfer, it to the production data
dictionary at Implementation Stage. See Chapter 7 .for more details on
documentation.
The physical data model (physical data base design) is created during the
Design Stage, and the data definition language statements (or copybooks) are
produced early in the Development Stage to support development and unit testing
of programs. These data structures are maintained throughout the rest of the
system life cycle under the control of configuration management.
5.5 Data Management Plan Topics Relating to Physical Data Base Design
Topics in the Data Management.PI an that cover physical data base design are
included below.
Design Stage
Logical Data Model Revision
Physical Data Base Design
Design Data Dictionary
Development Stage
Data Structures for Programming Support
Data Structure Revision Approach
Implementation Stage
Testing Support . ..
Life Cycle Methodologies arid Tools
(Supports physical data base design activities)
Data Security Strategy
(Requirements are supported by the physical data base design)
Plan for Physical Flow of Data
(Supported by the physical data base design)
Testing Support Plan
(Supported by the physical data base design and DBMS)
f Performance Validation Plan
31
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 6
DATA STEWARDSHIP RELATED ACTIVITIES
6.1 Introduction
If. you are the individual responsible for data management, you should
create a plan to determine and document data stewardship for the data your
system will collect. This plan should be done early in the Concept Phase, since
.several concept activities implement data stewardship.
6.2 Background
The Office of Solid Waste and Emergency Response uses thousands of data
elements while performing the missions assigned to it. The flow of OSWER
program data is often lengthy and complex, as data is collected at the state and
regional levels, then forwarded to OSWER 'for analysis, with the results being
disseminated to Congress, other agencies, the states, and regions. Managing the
complex activities, responsibilities and relationships which arise from these
data flows requires a method of determining which organizations involved in the
data flows are responsible for which data-related activities.
6.3 Guidance Objectives
The data stewardship guidance presented ..in this chapter has the following
objectives:
a. Facilitate the assignment of responsibilities for data definition,
collection, processing, storage, and use, when systems and data bases are
being built.
b. Ensure data meets mission requirements by assigning accountability for
data stewardship.
Co Ensure that data definition, collection; processing, and storage
methods within OSWER systems conform to applicable guidance.
/
d< Facilitate data sharing and reuse by clarifying roles and
responsibilities involved in the definition, collection, processing,
storage, and use of data.
6.4 What Is Data Stewardship?
Within OSWER, organizations may share data, but do not individually own it.
OSWER defines data stewardship as the functions and responsibilities of an
organization that exercises programmatic control over data on behalf of OSWER.
Organizations that require data to be collected, processed, stored .or used in
support of OSWER1s mission have stewardship responsibilities. These
responsibilities include ensuring that:
32 May 31, 1988
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
1) Data is clearly defined and documented in compliance with
established directives.
2) Data that is collected is of sufficient quality to support OSWER's
missions.
3) Only data relevant to OSWER's missions -is collected.
4) Data is reused wherever appropriate within OSWER.
5) Data management practices for data under the organization's
stewardship conform to OSWER guidance.
6.5 Which Organization Is A Data Steward?
The data steward role is separate and distinct from the role of a project
manager. Data stewardship assignment parallels OSWER program management
responsibility. The same organization with overall responsibility for OSWER
program mission performance is responsible for ensuring that the quality data
required to support the mission is defined, collected, processed, stored and
presented in a timely and cost-effective manner. If a project involves support
of more than one program, then multiple data stewards may be involved for the
data elements used by the project's system. Exhibit 6-1 provides an overview of
the data steward role.
6.6 What Are The Rights and Responsibilities of a Data Steward?
Data Steward Rights. The data steward organization appoints an individual
to execute its rights within the data stewardship guidance. The data steward
individual sanctions or approves the definition, collection, processing, storage
and use of data under his/her stewardship. This individual also has the right
to designate the data definer, the data collector and the custodian (see section
6.7).
Data Steward Responsibilities. The data steward organization is
responsible for ensuring that all the functional data roles are performed
adequately to provide data of sufficient quality to support OSWER program
missions. Other responsibilities include ensuring that only data relevant to
OSWER missions is collected, that data is reused wherever appropriate, and that
data is clearly defined and documented in conformance with EPA and OSWER
guidance. The steward organization appoints an individual to execute its
responsibilities within the data stewardship guidance.
6.7 What Are Functional Data Roles?
Exhibit 6-2 summarizes the function of each data role. A longer discussion
of each functional data role follows.
a. Definer.
The organization or function responsible for determining and
documenting the name, description, and other attributes of data required to
support an OSWER mission is the data definer. The data definer prescribes the
derivation rules and the formats to be used for data derived from other data
33
-------
EXHIBIT 6-1: THE DATA STEWARD ROLE
Data Steward
Organization
Appoints
CO
Designates
Definer
Collector
Data Steward
Primary User
Custodian
Sanctions
Ancillary
User(s)
Rights: Sanction definition, collection, processing, storage, and use of data.
Designate data definer, collector, primary user, and custodian.
Responsibilities: Ensures functional data roles are performed adequately to provide data of
sufficient quality to support OSWER program missions.
-------
EXHIBIT 6-2: FUNCTIONAL DATA ROLES
ROLE
FUNCTION
CO
in
Definer
Collector
Primary User
Custodian
Ancillary
Uscr(s)
Defines data requirements, meaning, content, and documents the needed data in "N
compliance with OSWER guidance. J
Collects data required by OSWER missions.
Uses data to perform mission functions. The primary data user is often the
data definer, as well. Supports testing during the systems life cycle.
Responsible for supporting the storage and processing of data. Ensures the physical
integrity of data used to support OSWER missions while providing access to the users.
Implements changes.
Uses data defined and stored to support the missions of others. The Ancillary user
requests appropriate approval from the Data Steward for accessing and using the data
sought. .
S
t
\
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
elements. The data definer may or may not be the same person or organization as
the collector. Usually, the data steward designates the primary user(s) of data
designated as the data definer. When multiple functions or organizations use
the same data to support important OSWER program functions, a joint data
definition effort will be organized. The data definer is responsible for
ensuring that OSWER and EPA data administration standards are met when defining
and documenting data.
b. Collector.
The organizations or functions responsible for collecting the values
of data required by an OSWER program are called the data collectors. The data
steward organization designates data collectors to direct and manage collection
of data for an OSWER program, after consulting with the collector organization
to ensure it can assume this role. The data collectors may not terminate the
collection of data without the authorization of the data steward organization.
c. Primary User(s).
The organization or function with the most important requirement to
collect, store and process data to perform a current or future OSWER mission
function is the primary user. Usually, the data steward organization designates
the primary user as the definer of the data the organization requires. Primary
user organizations also support user testing of systems during the life cycle.
If it .is impossible to select a single primary user organization from among
several users of the same data, then a joint data definition effort will
probably result. The primary user organization is often, although not always,
the data steward organization.
d. Custodian.
The organization responsible for storage and processing of data is the
data custodian. The functions carried out by the custodian include those which
have traditionally been performed by ADP organizations. That is, the custodian
has physical custody or direct control of the data, software and other ADP
components used to store, process, communicate and present data. The custodial
role ensures the physical integrity of data, and safeguarding the storage media.
During the life cycle, the custodial role may be assigned to the project team
until the data base (and system) is turned over to the operating organization
during the Implementation Stage. As small computers have become more common, it
is not unusual for custodial responsibilities to be split between different
organizations, or to be assumed directly by non-ADP professionals within the
primary user organization.
e. Ancillary User(s)
The ancillary users use data to perform OSWER functions, and report
results to management, the Congress, and to others outside the agency. Unlike
the primary users, ancillary users must rely upon others to define and allow
them access to the data.
36
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
6.8 Implementing Data Stewardship
During the Concept Phase, you should contact the OSWER Data Administrator
and discuss the scope of your project's application and the general types of
information you anticipate the application will need. If you and the OSWER data
administrator determine that multiple data steward organizations will be
involved in a project, plan to involve all of these organizations beginning in
the Concept Phase. Since implementation of stewardship at the data base or
project level is not feasible when multiple data steward organizations are
involved, record the steward organization for each data entity you identify
during Concept Phase. While there may be multiple data stewards for the data
collected by a system, there can be only one data steward per element of data.
Enter this stewardship information in the requirements data dictionary during
the Definition Stage. Ensure that data definers, primary users, collectors, and
custodians are recorded for all data elements.
As noted in Exhibit 6-3, when one organization defines, collects, and uses
a file, that organization is the probable data steward. However, when you find
data definition, collection, and use split between multiple organizations you.
face a more difficult problem when determining stewardship. Line 2 of Exhibit
6-3 provides an example of multiple organizations collecting and using the same
data file. When this occurs either of the primary user organizations may be
selected as the data steward, but not both. Data stewardship of data cannot be
shared between organizations. A single data steward organization must be
selected for clear accountability.
If your project is developing a high-impact system, you are likely to find
that several data steward organizations will be involved. For instance, the
application may collect enforcement data at the request of OWPE, and clean-up
data at the request of OERR. Be sure that both offices understand their
stewardship responsibilities, and work with them to provide the data they
require.
Use data dictionaries to save time when recording stewardship information.
For instance, if one steward organization sponsors an entire data base, then
record the stewardship information as part of the data base dictionary
documentation, and use the dictionary reporting facilities to link this
information to all data elements in the data base.
Data Stewardship Implementation Activities
Step 1 - During the Concept Phase identify the organizations that will likely be
the data steward organizations for the data your project will require.
Step 2 - Have the data steward organization appoint a data steward individual to
assume data stewardship responsibilities.
Step 3 - Work with the person appointed to determine the primary data user and
assign a data definer or definers for the project. Consult with the OSWER data
administrator. Involve the data definer(s) in identifying data entities as your
project team performs conceptual data modeling.
37
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Step 4 - After you have completed your data entity 11st (see Chapter 4), have
the data steward assign specific responsibility for defining each data entity's
meaning in mission terms. This same data definer will also define data elements
to describe the data entity more fully when the Definition Stage is reached.
Record the name of the data definer and data steward organization for each data
entity in the System Concept and in the definitions of data entities you create.
Step 5 - During the Definition Stage, record the data steward and the data
definer organizations for each data entity in the requirements data dictionary
and the detailed data requirements. Design your data dictionary entries so that
dictionary users can learn the data steward and definer for each data element,
too.
Step 6 - During the Definition Stage, determine the function and organization to
collect values for each data element; record the data collector in the
requirements data dictionary.
Step 7 - During the Design Stage, have the data steward name the organization
that will support technical operation of the system and data base once it is in
a production mode, and designate that function or organization as the custodian
for all data elements in the data base. Record this information in the design
data dictionary along with the steward and definer information you added to the
requirements dictionary earlier.
38
-------
EXHIBIT 6-3
DETERMINING DATA STEWARDSHIP
REVIEW FUNCTIONAL DATA ROLES TO DETERMINE PROBABLE DATA STEWARDS
DATA OBJECT
DATA
DEFINER
DATA
COLLECTOR
PRIMARY
USERS(S)
NON-PRIMARV
USER(S) CUSTODIAN
PROBABLE
DATA STEWARD
File I
ORG A
ORG A
ORG A
ORG none
ORG C
ORG A
CO
File 2
Data Element V
ORG A.B
ORG B
ORG D
ORG D
ORG A.B
ORG B .
ORG D.E
ORG A.O
ORG C
ORG D
OHU A or B
(not tiotli)
ORG B
Data Element Z
ORG O
ORG 0
ORG D
ORG A
ORG C
ORG D
Data Base X
ORG A
ORG A
ORG A
ORG B.D
ORG C
OKG A
ORG A = OSWER program office A
ORG B = OSWER program office B
ORG C = EPA data processing organisation C
ORG D = EPA regional office 0
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 7
DATA DOCUMENTATION ACTIVITIES
7.1 Introduction
This chapter describes the essential data documentation activities that
must be performed during the system life cycle. While documentation is required
of other life cycle products, such as the System Concept, this paper covers only
the essential documentation of data requirements, physical data base designs,
and production data base structures. The documentation must be completed to
support later phases of the life cycle, reduce maintenance costs, and provide an
audit trail from requirements to production data bases. Careful management of
this documentation will allow you to track your project's data requirements
through to the production data base.
7.2 Data Documentation Products
The OSdER Systems Development Life Cycle Management Guidance specifies that
project teams document detailed data requirements, physical data base designs,
and production data bases in data dictionaries. Each of these three data
dictionaries serve different purposes. While paper or a single software system
can hold all three of these dictionaries, the dictionaries should be created
discretely.
Dictionary Type Role
Requirements Dictionary Contains High Level and Detailed Data
Requirements
Design Dictionary Contains Physical Data Base Design
Production Dictionary Contains Production Data Base Design
7.3 Requirements Data Dictionary
The requirements data dictionary is produced by data modeling activities
that take place during the Concept and Definition Phases of the project.
Metadata recorded about data entities and data elements include their names,
programmatic definitions, purpose in the OSWER programs, data steward, data
definer, and source. Precise guidance regarding the metadata to insert into the
requirements data dictionary can be obtained by contacting the Office of Program
Management and Technology Information Management Staff.
Many automated software development tools provide good support for the
requirements data dictionary. If automated tools have been used to perform data
modeling, then some of the documentation in the requirements data dictionary can
be loaded from the software development tool into the data dictionary.
Likewise, automated tools used to support physical data base design should
4Q May 31, 1988
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
contain metadata derived from your data modeling tool, or from the requirements
data dictionary. If the metadata used in each tool Is based upon the metadata
produced by the preceding tool, you will save time and effort while minimizing
the risk of inaccurate metadata being used to produce a data base design.
7.4 Design Data Dictionary
The design data dictionary is initially created in the Design Stage and is
maintained through the rest of the life cycle. This dictionary contains
detailed data documentation of the physical data base design (see Chapter 5).
It also contains programmatic and administrative information stored in the
requirements data dictionary earlier. Programmatic definitions of data elements
stored in the requirements data dictionary are entered into this data
dictionary, as the physical design information is entered. If software is used
to store and use metadata, then the same piece of software can sometimes be used
for both dictionaries. This is a project decision.
Technical information about the physical data base design is entered into
the design data dictionary. It describes the design of the physical data base
structures and the manner in which these structures are implemented in the test
versions of the data base. For this reason the design data dictionary contains
many more types of metadata than the requirements data dictionary. In addition
to data elements, the design dictionary contains documentation of data bases,
physical records, segments, data sets (or files), and' keys. Information such as
block sizes, data set allocation, and physical size limits are documented in
this dictionary. .
If your project is using a data base management system which is controlled
by an active data dictionary, then use the active data dictionary for the design
data dictionary. These active dictionaries contain only the metadata for the
data bases they control, and they can save you time and money during system
maintenance.
7.5 Production Data Dictionary
The production data dictionary is initially created in the Implementation
Stage to support system integration and acceptance testing. Metadata from the
last version of the design data dictionary is moved to the production data
dictionary at the beginning of the Implementation Stage. Exactly the same
metadata is in the last version of the design dictionary and the initial version
of the production dictionary. The only difference between them is that the
former describes a data base under development and the latter a data base in the
final steps of testing and implementation.
You should continue to maintain a design data dictionary to support testing
of new versions of the data base throughout the rest of the life cycle. Once
the production dictionary is created, the design dictionary contains the same
information as the production dictionary, plus proposed changes. If you use a
data base management system that is controlled by an active data dictionary, use
the active DBMS dictionary as the production data dictionary. The "move" of
metadata from the design to production dictionary can usually be done by simply
changing the status indicator on those dictionary objects being moved into
production.
41
-------
OSWER DIRECTIVE ff902B.UUa
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Once metadata is moved into the production data dictionary it is secured
against changes or deletion. Entries in the production data dictionary are
never written-over or deleted, but new versions are created that reflect any
changes. Old versions are archived to create a perpetual history of the
structure of the production data base. The production data dictionary is
maintained up to the point when the data base is no longer used. Once the data
base is slated for conversion to another data base, or archiving, the metadata
in the production data dictionary is archived along with the production data and
software programs.
7.6 Importance of Metadata Management
As technology has evolved, data dictionaries have taken on an increasingly
important role. Many of the most modern, relational data base management
systems require that a production data dictionary be loaded before the DBMS is
used. Now, system and data base design activities are being supported by
automated software tools that are also designed around data dictionaries of
their own. As the individual responsible for data management for your project,
you will have to plan and monitor your project's collection, use, and transfer
of metadata throughout the system life cycle. If you fail in this task, you
could delay the project, increase the cost of the project, or cause a data base
to be implemented that doesn't meet OSWER requirements. You would be wise to
consult with the OSWER data administrator concerning the availability of
metadata management software, including the availability of an OSWER-wide data
resource inventory, to support your project.
Exhibit 7-1 illustrates the use of data dictionaries during the system life
cycle.
42
-------
DATA DICTIONARIES DURING THE SYSTEMS LIFE CYCLE
INITIATION
Identify
Information
Need
Define
High Level
Data
Requirements
Archive
Data &
Software
EVALUATIO
Pre-t listing
Data
Sources
DEFINITION
Production
Data
Dictionary
Define
Detailed
Data
Requirements
Evaluate
Data Base
Production
Data Dictionary
Requirements
DaU Dictionary
METADATA
Requirements
Data Dictionary
Production
Data Dictionary
(Updated)
PRODUCTION
Design
Data Dictionary
Production
Data Dictionary
Operate
and Maintain
Data Base
Design
Physical
Data Base
Design
Data Dictionary
Design
Data
Dictionary
Design
Data Dictionary
(Updated)
. Production
Data Dictionary
Create
and Test
Data Base
Convert
Production
Data
User
Acceptance
Test
Data Base
DEVELOPMENT
IMPLEMENTATION
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
CHAPTER 8
TERMS AND REFERENCE MATERIALS
8.1 Glossary of Terms
This chapter defines key terms used 1n OSWER's System Life Cycle
Management Guidance and this practice paper.
TERM
Access
Approval
Archive
Audit
Baseline
Change
DEFINITION
The operation of viewing or copying (including extracting)
data.
An examination of life cycle products, and the results of the
project review process, by OSWER program management. The
approval process has three purposes: first, to confirm the
results (i.e., the concepts, products, and management
direction) of life cycle efforts to date; second, to approve
continuation with the next stage of the life cycle; and third,
to confirm the continued commitment of resources to the
project. The OSWER life cycle model requires formal approvals
at the end of the Initiation and Concept phases, and the
Definition, Design, Development, and Implementation stages.
The third stage of the Operation phase, and the final stage o*
the system life cycle. Its purpose is to terminate the
operation of the system in an orderly, planned manner, ensuring
that software and data are properly archived or incorporated
into other systems.
A standard-oriented examination of the products and related
documentation contained in a baseline to assure that they are
complete, clearly presented, and internally consistent. The
OSWER life cycle provides for five audits; Concept, Definition,
Design, Development, and Operational. Any audit may be
repeated as necessary.
The set of life cycle products and other related documentation
which officially comprise the system at a given point in time.
The OSWER life cycle model provides for five baselines:
Initiation, Definition, Design, Development, and Operational.
The products contained in each baseline are always reviewed
prior to inclusion in the baseline.
An alteration to the system or d.ta base(s) for maintenance or
performance purposes, without affecting the functionality or
structure of the system or data base(s).
44
May 31, 1988
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Change Control
Concept
Conceptual
Data Model
Configuration
Accounting
Configuration
Management
Custodianship
Data
A process for controlling modifications to a system. Change
control provides a review of requested modifications, and
consideration of their impact on a system, before they are
made; it also ensures that changes are made in a manner that
does not disrupt ongoing system operation.
The second phase of the system life cycle. This phase provides
a high level of functional and data requirements that relate to
an information management problem, and a comprehensive model of
a solution to meet the requirements.
A depiction of data requirements from an organizational
perspective. Corresponds to the conceptual schema of a three
schema -architecture as defined by the American National
Standards Institute. Entity relationship diagrams are often
used to depict the conceptual data model.
A process for maintaining system baselines, including adding
products to a baseline, denoting the components of each product
(referred to as configuration items), and monitoring and
recording the disposition of requested changes to the system.
A function which serves to systematically identify the items
that make up a "system, and formally control any changes or
additions to those items, in order to help maintain the
integrity of the system, and facilitate communication about the
system throughout its life cycle.
The functions and responsibilities of an organization, such as
an ADP organization, with physical custody of data that
supports the work of another organization, such as a program
office. For example, the custodian ensures the physical
integrity of the data and software under its control;
safeguards the media storing data; ensures the data is secure
from unauthorized access, change or destruction; makes data
accessible to users; and implements requested hardware or
software changes.
Representations of facts, concepts, or instructions in symbols
suitable for communication, interpretation or processing by
human or automated means.
Data
Administration
The management function responsible for the planning,
definition, organization, protection, and efficiency of data
and data bases within OSWER. The goal of Data Administration
is the cost-effective provision of data of sufficient quality
to support the OSWER mission.
45
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Data
Administration
Program
Data Attribute
Data Base
Data Base
Management
System (DBMS)
Data Collection
Data Definer
Data Dictionary
Data Element
Data Entity
Data Flow
A management initiative which includes policies, standards, and
procedures that increase an agency's knowledge and management
of the: composition of data, source of data, processing of
data, meaning of data, flow of data, and dissemination of data.
A successful Data Administration program will improve the
management of data by introducing procedures that address: data
standards, data requirements determination, data definition,
data acquisition or collection, data processing, data storage,
data usage (including sharing and access), and data disposal.
A characteristic of a unit of data such as length, value, or
method of representation.
A collection of interrelated data stored together with
controlled redundancy to serve one or more systems or
applications.
A software system facilitating the creation and maintenance of
a data base and the execution of computer programs using the
data base.
The recording
organization.
and capturing of data on behalf of an
The person or organization who determines the essential
qualities or meaning of data, and who prescribes and defines
procedures which -aggregate and refine data. This includes
describing the formatting of the resulting information to serve
a specific decision-making context.
A centralized repository of information about data, including
its meaning, relationship to other data, origin, usage and
format. A PIPS data dictionary standard is expected to be
approved late in 1988. Called an Information Resource
Directory System (IRDS), this standard specifies the
capabilities that should be offered for future data
dictionaries. . I
The smallest unit of data that has meaning in describing
information. A piece of data which would not be meaningful if
decomposed further.
A person, place, thing, concept, or event about which an
organization may store data. Data entities are to nouns as
data elements are to adjectives. That is to say that data
entities are the objects being described by data elements.
Entity is a synonym for data entity.
A depiction of the movement of information (data) between
processes. Data flows are a'component of data'flow diagrams
used by systems analysts to analyze application system
requirements.
46
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Data
Independence
Data Integrity
Data Life Cycle
Data Security
Data Steward
Data Store
Decision Paper
Definition
Definition and
Design
Design
Development
The property of a data base management system that enables data
to be processed independently of access mode, storage method or
arrangement* Data independence reduces the need to modify
application programs when data storage and access methods are.
modified.
The quality of data that exists as long as accidental or
malicious destruction, alteration or loss of data are
prevented. This results in preservation of data in its
intended format, length and contents while within a data base.
The data life cycle begins with the definition of data to
support new regulations or other program needs, and includes
strategic data planning, data standardization, and the methods
and standards during the collection, storing, accessing, and
archiving of data.
The protection of data against unauthorized disclosure,
transfer, modification or- destruction, whether accidental or
intentional.
See Stewardship.
A component of a data flow diagram that represents data at rest
or in storage.
A decision document presented to management. It summarizes
those aspects .of-the analysis and decisions of a given phase or
stage that are important to program management, and requests
approval to continue the project. The OSWER life cycle model
provides for Decision Papers to be prepared at the end of
Concept, Definition, Design, Development, and Implementation.
The Mission Element Needs Statement (MENS) also is considered a
Decision Paper.
The first stage of the Definition and Design phase. Its
purpose is to define specific, detailed functional and data
requirements for the system within the context of the System
Concept.
/
The third phase of the system life cycle, consisting of two
stages: Definition and Design. (See individual definitions of
each of these terms).
The second stage of the Definition and Design phase. Its
purpose is to produce detailed specifications for the system to
meet the functional and data requirements within the context of
the System Concept.
The first stage of the Development and Implementation phase.
Its purpose is to produce a system which is ready for
acceptance testing and suitable for implementation.
47
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Development and
Implementation
Domain
Enhancement
Entity
Entity
Relationship
Diagram
Evaluation
Implementation
Information
Information
Flow
Initiation
The fourth phase of the life cycle. Its purpose is to produce
a complete system, fully tested and available for use in normal
production mode. There are two stages in this phase:
Development and Implementation.
A set of all values that a particular data element may posses
in actual or potential usage.
A modification to a system that results in substantially
improved capabilities and, in some way, alters the structure of
the system. Other modifications that do not alter the
structure, are referred to as changes. Examples of
enhancements include the addition of new data elements,
changing the system (or a part of the system) to run in a
different software environment, and replacing data entry
screens to improve the user interface and/or improve
performance.
A person, place, thing, concept, or event that is of interest
to an enterprise. An entity is something about which we store
data. Examples of entities are: waste site, contract,
employee. An entity has various attributes, or data elements,
which should be recorded. Examples for the entity contract
could include contract-number, date, and obligation-ceiling.
Entity is a synonym for data entity.
A diagram- that depicts entities, their key attributes-(data
elements), and the relationships between entities. The entity
relationship .diagram is the most commonly used.data modeling
technique used today. Sometimes system developers refer to
entity relationship diagrams as ER diagrams or static data
models.
The second stage of the Operation phase. Its purpose is to
determine whether the system is effectively.meeting the stated
requirements, is operating efficiently and is effectively
managed.
The second stage of the Development and Implementation phase.
Its purpose is to produce a fully tested system'containing the
data needed at start-up, and to provide needed training to the
intended users.
Any set of data which has been aggregated by processing in
order to establish a specific meaning and serve in a
decision-making context.
A depiction of the flow of information between processes. An
information flow is another name for a high-level data flow.
The first phase of the system life cycle. Its purpose is to
define an information management problem within OSWER and to
determine whether resources should be committed to exploring
ways to address it.
48
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Initiation
Decision Paper
Life Cycle
Life Cycle
Management
Logical Data
Model
Maintenance
Metadata
Normalization
Operation
Phase
Privacy
A brief document identifying and describing an information
management problem. This document is prepared during the
Initiation phase.
See 'System Life Cycle11
The process of managing a system through its life cycle. As
practiced by OSWER, it is not a rigid process, but rather a
disciplined means for selecting and practicing the management
approaches and techniques that are most appropriate for a given
information management problem and/or system.
A depiction of the logical, or programmatic, data needed to
support an organizational mission. The components of a logical
data model include data entities and relations; data elements
or attributes; keys; secondary keys; and relationships, if data
entities are used. The logical data model is a more detailed
depiction of the conceptual data model of an organization. It
may correspond to the external schema as defined by the
American National Standards Institute.
The set of activities that keep systems and data bases in
operating condition. Maintenance also focuses on optimizing
the performance of existing systems and data bases, without
affecting functionality or the structure of the systems or data
bases.
Data about data, such as its definition or its physical
characteristics.
The process of reducing a logical data model (structure) to its
most basic, form, so that the data model is stable, flexible,
and without redundancy. A normalized data model is composed of
normalized data entities. A normalized data entity includes no
repeating groups or data elements among its attributes,
contains attributes (data elements) only about the entity being
described, and does not include attributes which are dependent
upon the key of another entity.
The fifth phase of the life cycle. Its purpose is to operate
the system in normal production mode, monitoring and
maintaining its performance, until the end of the life cycle,
and then to terminate operation. There are three stages in
this phase: Production, Evaluation, and Archive.
The major segments of the system life cycle. There are five
phases in the OSWER system life cycle: Initiation; Concept;
Definition and Design; Development and Implementation; and
Operation.
The right of individuals or organizations to constrain the
collection and use of data about themselves.
49
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Production
Project
Project
Execution
Project
Management
Quality
Assurance
Record
t
Review
Shared Data
Stage
The first stage of the Operation phase. Its purpose is to make
the system available to users, and make required changes and
improvements to ensure that it contains to address the
information management problem in a cost effective manner.
An organized effort to solve an information management problem.
In most cases, a project extends over the entire system life
cycle. In some cases a project extends only through the
portion of the life cycle that can be foreseen with confidence,
e.g., through Production if the timing for ceasing operation 1s
uncertain.
The set of activities which produce the concept, definition,
design, and production versions of a system.
The set of activities which monitor and control project
execution to ensure that they are performed effectively and in
accordance with applicable policies, guidances and practices;
and that its products solve the identified information
management need.
A function that ensures that all products of the life cycle are
substantively accurate and address the stated information
management problem. Quality assurance is accomplished through
the efforts or skilled professionals on the project team, and
through formal reviews.
A group of related
application program.
data elements treated as a unit by an
A formal examination of a life cycle product to verify that the
system as represented solves the specified information
management .problem. The OSWER life cycle model provides for
five reviews; Concept, Definition, Design, Development, and
Post- Implementation. Any review may be repeated as necessary
to ensure that all deficiencies have been fully and -adequately
addressed in the designated products.
Data stored that is created, accessed, updated, or deleted by
more than one organizational unit.
The segments of the system life cycle that occur within certain
larger phases. The OSWER system life cycle divides the
Definition and Design phase into two stages: Definition and
Design. The development phase is divided into two stages:
Development and Implementation. The Operation phase is divided
into three stages: Production, Evaluation, and Archive. The
phases Initiation and Concept are not divided into stages.
50
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Stewardship
System
System Concept
System Life
Cycle
Threshold
Analysis
Walkthrough
The functions and responsibilities of an organizational entity
that exercises control over data on behalf of OSWER.
Organizations that require data to be collected, processed,
stored or used in support of OSWER1s mission have stewardship
responsibilities. These responsibilities include ensuring
that: (1) Only data relevant to OSWER's missions is collected.
(2) Data that is collected is of sufficient quality to support
OSWER's missions. (3) Data is reused wherever appropriate
within OSWER. (4) Data is clearly defined and documented in
compliance with established directives. (5) Systems practices
under the organization's stewardship conform to EPA Data
Administration guidance.
An organized set of functions, data, procedures, hardware,
software, communications, and/or documentation which enables
OSWER to solve a specific information management problem. A
system need not necessarily be automated; but most instances of
life cycle management will apply to automated information
systems.
A high-level complete description of a system (including data,
processing capabilities, hardware, software and
communications). It is produced during the Concept phase and
serves as both a check on the validity and completeness of the
problem, and the basts for defining more detailed functional
and data requirements.
The evolution of a system from the 'initial identification of an
information management problem through system termination or
replacement.
The process of determining the appropriate review and approval
levels for an OSWER system project.
A highly-structured meeting to review the completeness and
quality of selected module(s) of the system, or of the entire
system. Walkthroughs are usually conducted by the project
team, often are attended by user representatives, and may be
held at any point 1n the system Hfe cycle.
51
-------
PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
8.2 Additional Reference Materials
8.2.1 Data Modeling References
o Entity Modeling. Ronald Ross, Database Research Group, Boston, 1987.
o Guide on Logical Database Design. Elizabeth Fong, et al., National Bureau
of.Standards Publication 500-122, 1985.
8.2.2 Physical Data Base Design References
o Computer Data Base Organization. James Martin, Prentice-Hall, Englewood
Cliffs, New Jersey, 1975.
o Data Base; Structured Techniques for Design, Performance, and
Management, S. Atre, A. Wiley Series, New York, 1980. ^
o Design and Strategy For Distributed' Data Processing, James Martin,
Prentice-Hall, Englewood Cliffs, New Jersey, 1981.
8.2.3 Data Documentation References
o Data Dictionary/Pirectory Systems, BeIkis Leong-Hong and Bernard Plagman,
John Wiley & Sons, New York, 1982.
o Data Dictionaries and Data Administration. Ronald Ross, AMACOM, New
York, 1981.
o Guide on Data Entity Naming Conventions, Judith Newton, National Bureau
of Standards Publication 500-149, 1987. . '
52
-------
OSWER DIRECT We Wi^cc.j*.^
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
Part 3: Practice Paper
Project Management Plan
January, 1989
-------
TABLE OF CONTENTS
Page
1. Practice Paper Purpose 1
2. Structure and Contents of Project Management Plan 2
2.1. Project Charter/Objectives 2
2.2. Life Cycle Adjustment 2
2.3. Project Team Oraanization .: 4
2.4. Project Budget 6
2.5. Project Reviews/Quality Assurance 7
2.6. Applicable Project Approvals 8
2.7. Benefit-Cost Analysis 9
2.8. Methodologies.and Tools 9
2.9. Workplan , 11
2.10. Procurement Approach 12
2.11. Configuration Management Plan '. 13
2/12. Documentation Standards 14
2.13. Security Approach 14
2.14. Conversion Approach 15
2.15. Installation Approach » 16
2.16. User Support Approach 16
2.17. Maintenance Approach 17
2.18. Operations Approach 18
3. Development and Update of the Project Management
Plan 19
3.1. Responsibility for Developing and Updating the
Project Management Plan 19
-------
?»vn L.':nlw'«V >. -.,--
TABLE OF CONTENTS (Continued)
Page
* 3.2 Format of the Project Management Plan 19
3.3 Evolution of the Project Management Plan Through
the system Life Cycle 19
3.4 Retention of Old Project Management Plans 22
4. Relationships Between Project Management Plan Topics .. 23
EXHIBITS
~s
2-1. Outline of Project Management .Plan 3
3-1. Evolution of Project Management Plan Through the
System Life Cycle 20
4-1. Summary of Relationships Between Project Management
Plan Topics » 24
4-2. Details of Relationships Between Project Management
Plan Topics . 25
Appendices
A. Detailed Outline of Project Management Plan 34
-------
OSWER DIRECTIVE #S023.GOa
1. PRACTICE PAPER PORPOSE
This practice paper constitutes a section of Part 3 of the
Office of Solid Waste and Emergency Response (OSWER) System Life
Cycle Management Guidance. It describes the Project Management
Planr a key document of the system life cycle. Every system
project is required to develop and use a Project Management Plan.
This practice paper describes the structure and content of the
Project Management Plan, and its evolution through the system
life cycle.
OSWER places great emphasis on the Project Management Plan
because of the intrinsic difficulty of managing system projects,
especially projects for systems that support more than a handful
of individuals. Rigorous development and use of the Project
Management Plan will help ensure that important issues regarding
the approach to the project are carefully considered and the
decisions are documented. The Project Management Plan also helps
to communicate the approach and coordinate the approach across
all project team members, and to clearly measure progress in
completing the project.
The topics addressed in this practice paper include:
o The structure and content of a complete Project Management
Plan;
o Responsibility for. preparing and updating the Project
Management Plan;
-o How the Project Management Plan evolves through the system
life cycle; and ,
o How the components of the Project Management Plan relate
to each other and to the other products of the system life
cycle.
The Project Management Plan serves several important purposes
in support of the system life cycle:
o Helps ensure that important issues are purposefully
considered and that key decisions are clearly documented;
o Helps support the coordination of various organizations
and individuals involved in a system project, by providing
a single known source of information regarding the project
approach and the role of each organization/individual; and
o Provides a basis for measuring progress through the system
life cycle, and the potential impact of any changes to the
'approach for'conducting the project.
-------
This practice paper is not intended as a primary means for
training project managers. Rather, it describes an approach for
clearly documenting certain project management topics and
decisions of importance to OSWER. It should also be noted that
although the Project Management Plan references certain
characteristics of an information system (e.g., software tools,
security), the Project Management Plan describes the logistics
for the project. It does not serve as documentation of the
system requirements, design or other features of the system
included in other formal system documentation.
2. STRUCTURE AND CONTENTS OF PROJECT MANAGEMENT PLAN
The Project Management Plan should use the same basic
structure for all systems projects. This structure is presented
in Exhibit 2-1. All topics shown in Exhibit 2-1 should be
included in the plan; however, the level of detail at which each
is discussed should be tailored to the individual project.
No specific format is required for most topics; however,
certain specific information should be provided. This section of.
the practice paper, identifies the information to be provided for
each topic of the Project Management Plan, and suggests specific
formats or presentation techniques where appropriate. Appendix A
provides a more detailed outline of the complete Project
Management Plan.
2.1. Project Charter/Objectives
Every system project should have a clear charter,
describing the objectives of the project and certain other key
project attributes. This section of the Project Management Plan
provides the overall context for the other sections of the Plan.
It summarizes the following information from the Project
Initiation Decision Paper: :
o The information management problem to be solved,
t
o The scope of the problem in terms of OSWER programs and
organizations,
o The timeframe for solving the problem, and
o The organization(s) and individual(s) that serve as
programmatic sponsor for the project.
2.2. Life Cycle Adjustment
Parts 1 and 2 of this Guidance describes specific sequence
of life ' cycle phases and stages, a sequence that applies to the
entire system. For some projects, it may be desirable to adjust
-------
UN. I I - - ' -"
EXHIBIT 2-1: OUTLINE OF PROJECT MANAGEMENT PLAN
TOPICS
Project Charter/Objectives
Life Cycle Adjustment
Project Team Organization
Project Budget
Project Reviews/Quality Assurance
Applicable Project Approvals
Benefit-Cost Analysis
Methodologies and Tools
Workplan
Procurement Approach
Configuration Management Plan
Documentation Standards
Security Approach
Conversion Approach
Installation Approach
User Support Approach
Maintenance Approach.
Operations Approach
-------
the life cycle, such as by combining certain stages, or by
dividing the system into different modules, each with its own
schedule for progressing through the life cycle. This section of
the Project Management Plan is extremely important, because it
establishes the framework for many other sections, particularly
the project Workplan. This section describes any significant
planned adjustments to the conventional system life cycle
described in Parts 1 and 2 of this Guidance, and the reasons for
such adjustments. Examples of the types of adjustments that
should be included in this section are:
»»-
o The consolidation of portions of two or more stages,
such as the generation of software (part of the
Development stage) during the Design stage,
o Partitioning the project or system into modules or work
packages (usually done during the Concept phase), with
different life cycle schedules for one or more modules,
o Phased development of the system or data base using
multiple life cycles one to provide basic system
capabilities, and subsequent cycles to provide expanded
capabilities through the planned replacement of major
portions of the system,
o Iterative cycling through portions of the life cycle, as
is 'often the case in the development of an expert
system,
o Consolidation of two or more system life cycle products,
including consolidation of System Decision Papers, and
o Elimination of any system life cycle products.
2.3. Project Team Organization
This section describes how the project team will be
organized in terms of the specific organizations and individuals
who will participate actively in the project. This section is
particularly useful for large projects, with many participating
organizations and individuals. The Project Manager may use this
section of the Plan as a stand-alone document, distributing it to
all participating organizations (including contractors) and
individuals to improve project coordination.
Specific information contained in this section should be
documented using an organization chart, as well as other
applicable techniques, and includes:
o Identification of the Project Manager, his/her current
home organization, and any assignment of this individual
to another organization (e.g., detailing to another
office) to accomplish his/her role as project manager;
-------
OSWEP DIRECTIVE *?3023.DOa
o Identification of any supporting organization structures
that will serve in a project management role, such as
boards and advisory committees, and the roles and
authorities of such organizations. These organizations
may be unique to the system, or may be standing
organizations with management responsibilities for
systems affecting a designated program;
o Description of project staffing, including Agency
personnel and contractor support. The home organization
and percentage and duration of assignment and for each
Agency team member should be clearly identified.
Specific contractor organizations should be identified
as soon as is practical. Total contractor staff
assigned to the project, and key contractor personnel
should be identified as well. The roles of each member
of the project team should be clearly identified on a
person-by-person basis or, for very large projects, by
identifying the specific sub-team to which each member
is assigned. Experts in programmatic or technical
subject matter of particular importance to the project
should be clearly identified.
o Description of the structure of the project staff.
reporting to the project manager, including the
identification of any sub-teams (if applicable) and size
and team leaders for each team;
o Identification of the data steward for the project, or
multiple stewards if appropriate, for different types of
data;
o Identification of individual organizations that have.an
interest in the system and are not directly represented
on the project team, but which will be informed of major
milestones and decisions through the distribution of
required system decision papers and other materials as
appropriate. Examples include:
.. Individual regional waste management program
organizations,
Office of Information Resources Management,
National Data Processing Division - NCC and WIC,
Individual .regional ADP organizations,
Individual State waste management program
organizations, and
Office of the Inspector General;
-------
OSWER DIRECTIVE #332E.W
Identification of the members of the Change Control
Board, and the authority of the Board (i.e., a
decision-making body or an advisory body to the Project
Manager).
OSWER requires the use of block diagrams, or similar
techniques, to illustrate the project team organization.
Multiple diagrams should be used to illustrate team structures
that are expected to change throughout the life cycle.
A separate System Life Cycle Management Guidance Practice
Paper entitled 'Project Participation and Coordination1 provides
suggestions for identifying the organizations who should
participate in each system project. Of vital importance, the
contents of the Project Management Plan should be coordinated
with all the organizations that will be involved in the project.
The level of commitment of Agency staff to the project, and their
commitments to other assignments, must be agreed on by the
Project Manager and each participant's supervisor.
2.4. Project Budget
This section identifies the approved resources to be used
to accomplish the project, the source of funding for all
resources in terms of organizational entities (e.g., allowance
holders and ^suballowances), and the accounting methods and
procedures that will be used to monitor the project budget. The
Project Budget section of the Project . Management Plan is
particularly important because it describes a commitment of
resources, and not just a need for resources. The project budget
is broken out for each phase and stage, and identifies the
resource level and cost of the following types of .resources, as
applicable:
o EPA staff,
o Contractor services,
o Equipment purchase or lease,
o Equipment maintenance, /
o Site preparation (e.g., to accommodate ADP equipment),
o Software package(s) purchase or lease,
o Supplies,
i
o Computer timeshare (internal to EPA such as ase of the
National Computer Center mainframe, and external
services)
o Other costs
-------
OSWEn D!r£C::;'.£ i;Z~^.-^.
For some projects, the Project Budget may also serve to
indicate the need for additional resources the difference
between the resources needed for each phase and/or stage and the
commitments received to date.
OSWER places particular emphasis on effective monitoring
and control of project resources and budgets. For each of the
above resources, this section of the Project Management Plan
identifies the procedures and tools to be used to track the
expenditure of project resources against the budget provided by
each funding source.
Of particular note, the Project Budget (together with the
Benefit-Cost Analysis) serves as the source of cost information
used to determine the appropriate level of review and approval
for the project (i.e., Threshold Analysis).
2.5. Project Reviews/Quality Assurance
This section identifies the individual formal project
reviews and other quality assurance activities to be conducted
during the system life cycle. Project reviews are a key step in
each phase and stage of the life cycle they provide feedback
to the project team, and are advisory to the project approval
authority who will be asked to approve the continuation of the
project. (The required reviews, and technique for determining
who should conduct them (i.e., Threshold Analysis), are described
in the practice paper on 'System Life Cycle Reviews and
Approvals'.)
Some of the information contained in this section of the
Project Management Plan will be developed by the Lead Reviewer
for the project, and should be provided to the Project- Manager.
Specific information contained in this section includes:
o Identification of the applicable 'threshold', or
organizational level for conducting required reviews.
For a Level I system, also designates the criteria that
result in the Level I classification;
/
o Identification of the specific formal project reviews to
be conducted in each phase and stage, and approximate
schedule. The number of reviews and schedule should be
structured to reflect any adjustments to the system life
cycle.
o Identification of the specific organizations and
individuals who will participate in each review;
designated individuals should be independent of the
project team;
o Description of how the reviews are to be conducted, and
the approach/procedure to be used to document the
results of reviews; and
-------
o Drawing from the project Workplan, identification of
other activities to be conducted to confirm the
programmatic and technical findings and recommendations
of the project team (e.g., system design walkthroughs
and presentations, circulation of life cycle products to
user and other organizations for comment, independent
validation and verification (IV&V), acceptance testing).
To ensure that the project team will effectively solve
the information management problem, these other
activities are strongly encouraged. Reviews should not
be limited to only the formal reviews and specified for
each phase of the life cycle.
2.6. Applicable Project Approvals
This section identifies the individual formal project
approvals to be obtained during the system life cycle. OSWER
requires that every project be approved at the end of each phase
and stage of its life cycle to ensure that it will solve the
information management problem, within an acceptable timeframe,
and with reasonable resources. (The required approvals, and
technique for determining the approval authority (i.e., Threshold
Analysis), are described in the practice paper on 'System Life
Cycle Reviews and Approvals'.) Specific information contained in.
this section includes:
o Identification of the applicable 'threshold', or
organizational level for providing the required
approvals. For a Level I system, also designates the
criteria that result in the Level I classification;
o Identification of the specific approvals to be obtained
in each phase and stage, and approximate schedule. The
points of approval and approval schedule should be
structured to reflect any adjustments to the system life
cycle.
o Identification of the specific organizations and
individuals who will participate in the approval
process, and the means to be used to present system
decision papers and other life cycle 'products (as
appropriate) to the approval authority;
o Description of the approach/procedure to be used to
document the results of each requested approval;
o Drawing from the project Workplan, identification of
other approvals tc be secured by the project, in
addition to those identified in Part 2 of the OSWER
System Life Cycle Management Guidance;
-------
OSWER DIRESTtVS ^222-
2.7. Benefit-Cost Analysis
This section provides a summary of the system benefit-cost
analysis. This analysis is first presented in the Initiation
Decision Paper as an initial rough estimate of project scale, and
a comprehensiver detailed analysis is conducted during the
Concept phase and is contained in the System Concept document.
The Benefit-cost analysis presented' in this section of the
Project Management Plan draws on these life cycle products for
both benefit and cost information. As the system evolves through
the life cycle, this analysis must be updated. The current
perspective of benefits and costs is documented in detail as a
refinement to the System Concept (contained in the Initiation
Baseline) and is documented in summary form in this section of
the Project Management Plan. Specific information contained in
this section includes:
o Analytic methodology and major assumptions regarding
program direction, information management technology,
resource availability, and/or other issues as
applicable;
o System benefits:
Program effectiveness (quantified as specific
measures of improvement if possible),
One time monetary benefits,
Recurring/annual monetary benefits;
o System costs:
Initial investment (e.g., Initiation phase through
Implementation stage),
Recurring/annual costs,
Total system life cycle costs;
o System payback period; and
o Sensitivity of estimated benefits and costs to
identified assumptions.
Of particular note, the costs documented in this section of
the Project Management Plan, together with the Project Budget,
serve as the cost information needed to conduct the Threshold
Analysis for project reviews and approvals.
2.8. Methodologies and Tools
This section provides a summary of the methods and tools
selected to conduct the activities of the system life cycle.
-------
Each phase and stage of the life cycle should be conducted using
an appropriate set of systems analysis and development methods
tools. This section of the Project Management Plan identifies
the methods and tools to be used, and also describes how the
methods and tools will work together. It also describes how the
tools will be used to produce the required documentation and
other products of the life cycle, and any adjustments to the
products (per the outlines contained in Part 2 of this Guidance).
This section draws from the System Concept the initial
selections of methods and tools. These selections are confirmed
during subsequent phases and stages, and any new or changed
selections are documented in summary form in this section of the
Project Management Plan. Examples of the types of methodologies
and tools identified in this section of the Project Management
Plan include:
o Techniques and software tools to support system
requirements analysis (e.g., system prototyping),
o System analysis and design methodologies (e.g., Yourdon
structured analysis, application generators),
o Techniques for data analysis (e.g., entity-relationship.
analysis),
o Computer-aided software engineering (CASE) tools,
6 Programming languages (e.g., COBOL)
o Programming aids and debugging tools (e.g., OPTIMIZER
III),
o Communications software (e.g., CICS, Kermit),
o Data base management software (e.g., ADABAS),
o File management and configuration management software
tools (e.g., TIP Repository),
o Project management tools (e.g., TELLAPLAN,'SUPERPROJECT)
and;
o Word processing software (e.g., WORDPERFECT).
Specific information contained in this section for
individual selections of methodologies and tools includes:
o Identification of methodology or tool,
o Training/other special support required, and
o Procurements needed for acquisition and/or support.
10
-------
OSWER DIRECTIVE #£023.COa
As illustrated in Exhibit 3-1, the selections for each
phase and stage are finalized at the end of the immediately
preceding phase/stage.
2.9. Workplan
This section describes in detail the logistic;; for
conducting the project. It is structured to parallel the .
individual phases and stages of the system life cycle. The
workplan describes the specific tasks for conducting the project,
noting the relationships between tasks. For projects that are
very large, complex, or on a very tight schedule, the workplan is
particularly important it identifies the 'critical path1 of
activities that are instrumental to the success of the project.
The workplan also identifies resources for each task, serving to
clearly allocate the resources provided in the Project Budget.
The Workplan is most detailed for the immediately upcoming
phases or stages and, as illustrated, in Exhibit 3-1, is examined
in detail and confirmed for each phase or stage prior to
initiation of work in that phase or stage. The Workplan contains
the following information for each phase/stage:
o Identification of all project activities, and work-
breakdown of activities into more discrete tasks as
appropriate;
o Identification of all . products, and mapping of
activities/tasks to products;
o A schedule (i.e., start and completion dates for each
activity/task) documented in the format of a Gantt
chart, including the schedule for required formal
reviews and approvals *;
o Agency staff and contractor assignments to each
activity/task *;
o Level of resources/funding for each activity/task and/or
-life cycle product *;
o Schedule relationships/dependencies between activities
and/or tasks, including dependencies with regard to
activities/tasks for other phases or stages *; and
o For very large projects, where the system is divided
. into modules or work packages, it will be helpful to
prepare a high level workplan integrating the project
tasks across modules, and a more detailed workplan for
each module. *
o For projects involving a procurement of hardware,
software, or support services, one part of the Workplan
11
-------
OSWER DIRECTIVE #9028.00a
should be devoted to the activities of the "Procurement
Approach" for the project.
For those items denoted above with an asterisk {*), it is
suggested that automated project management tools be used to
develop and document the corresponding portions of the project
Workplan. The project Workplan also identifies the approach to
be used for project status reporting, including procedures,
report content, frequency, and assignments of personnel to
perform status reporting.
2.10. Procurement Approach
This section summarizes the means to be used to acquire all
contract support services, to acquire any needed hardware,
software, and communications capabilities that are not currently
installed at needed locations, and to obtain support from other
government organizations (e.g., interagency agreements). Most
projects include at least one significant resource acquisition,
and the Procurement Approach helps ensure that the needed
resources can be obtained and are available at the time they are
needed.
The Procurement Approach should be complete for all stages .
through Production by the end of the Concept phase if possible,
and no later than the end of the Definition stage, to ensure that
adequate lead time is available to acquire needed resources. It
is important to prepare this section of the Project Management
Plan even if the project intends to acquire resources through an
existing contract. Specific information contained in this
section includes:
o Resources to be acquired through existing OSWER
contracts:
Resource identification (e.g., specific hardware,
software, communications or service),
Contract identification,
» /
Planned acquisition date, and
Lead contact person on project team;
o Resources to be acquired through existing contracts of
other Agency offices:
Resource identification
Contract identification,
Planned acquisition date, and
Lead contact person on project team;
12
-------
CSWER DIRECTIVE #902S.OOa
o Resources to be acquired through new procurements:
Resource identification/
Planned procurement award date,
Scope of procurement anticipated (procurement for
single project/system or a procurement to support
multiple projects/systems),
Type of procurement (e.g., full and open
competition, limited competition, sole source
award),
Lead contact person on project team,
Lead contact person(s) at other Agency organiza-
tion(s) providing procurement assistance, and
[Note __ that the wofkplan (tasks, milestones,
schedules, staffing) for accomplishing all
activities needed to complete the procurement is
included in the "Workplan" section of the Project
Management Plan;
o Support to be acquired from other EPA offices and
government organizations (e.g., General Services
Administration):
Organization identification,
Type of support needed,
Planned start date for support,
Lead contact person on project team, and
Lead contact person at support'organization;
>
-- Tasks and task schedule for establishing needed
agreement.
Some of the content of the Procurement Approach may be
sensitive, and should be maintained and stored in a manner to
prevent disclosure t to contractors in advance of the proper time
for formal notification of upcoming procurement "actions.
i
2.11. Configuration Management Plan
This section describes the organization, procedures and
tools used to identify, monitor and control the configuration of
the system. Configuration Management is an important function
within the OSWER system life cycle it serves to ensure the
integrity of the system throughout its life cycle. The
13
-------
Configuration Management Plan describes in detail how the project
will conduct Configuration Management. Specific information
contained in this section includes:
o Identification of Configuration Manager,
o Identification of Change Control Board organizations
represented and individual members, and authority of the
Board.
o System baselines (identification/index of configuration
items),
o Change request review and approval procedures,
o Configuration status accounting procedures, and
o Software control procedures.
Some Configuration Management "Plans may be quite long, and
can be maintained as a stand-alone document that is referenced in
the Project Management Plan.
A separate System . Life Cycle Management Practice Paper.
entitled 'System Configuration Management1 describes OSWER's
practice of configuration management, and explains in more detail
the content of the Configuration Management Plan.
2.12. Documentation Standards
This section identifies the standards to be used in
producing the system documentation required in each phase and
stage of the system life cycle. Standards are particularly
important when contractor staff are preparing system
documentation, because these standards are a key basis for
determining whether the contractor has delivered adequate
documentation.
This section includes the identification of specific OSWER
standards, . standards prescribed by the Agency, .and standards
adopted from other organizations, such as the Federal Information
Processing Standards (FIPS) issued by the Bureau of Standards,
Department of Commerce. In the absence of a mandatory standard
for a system life cycle product, this section should identify a
comparable product(s) produced by other projects that will serve
as a model for the current project.
f
2.13. Security Approach
This section provides a summary of the security
requirements and security features of. the system. It is included
in the Project Management Plan to provide an overview of security
needed to prepare and review the Project Workplan and other
sections of the Project Management Plan. This section draws from
14
-------
OSWER DIRECTIVE *302a.QCa
the System Concept, Detailed Functional Requirements, Detailed
Data Requirements, and System Design to provide a summary of the
system and data security requirements and the system features
that meet these requirements. Specific information presented in
this section at a summary level includes:
o Functional security requirements,
o Data security requirements, including identification of
confidential or sensitive information,
o Project team organization to develop and support
specific security features and capabilities (if
applicable),
o Hardware and facilities access security measures,*
o Software and communications security measures,
o Data base security measures,'
o Procedural measures (e.g., procedures for handling
confidential or sensitive input documents and system
outputs), and
o Software and data base, backup and recovery measures.
2*14. Conversion Approach
This section draws from the System Concept, Detailed
Fxmctional Requirements, Detailed Data Requirements, and System
Design to provide a summary of the data to be converted from
existing systems and data bases, conversion activities and
procedures, and organizations responsible for accomplishing the
conversion. It is included in the Project Management Plan to
provide an overview of the conversion approach needed to prepare
and review the Project Workplan and other sections of the Project
Management Plan. Specific information presented in this section
at a summary level includes:
y
o Identification of major types of data to be converted,
including conversion of data currently maintained in
hardcopy form using manual procedures as well as the
conversion of data currently maintained by automated
systems;
o Identification of the following for each type of data to
be converted:
Source and location of data, .
Anticipated data quality problems,
15
-------
CSWER DIRECTIVE
Organizations) responsible for data cleanup,
Organizations) responsible for planning and
conversion,
Conversion schedule, and
Reference to specific sections of system
documentation describing detailed conversion
procedures and software.
2.15. Installation Approach
This section draws from the System Concept and System
Design to provide a summary of the logistics for installing the
system and data base in .the production environment. It is
included in the Project Management Plan to provide an overview of
the installation approach needed to prepare and review the
Project Workplan and other sections of the Project Management
Plan. Specific information presented in this section at a
summary level includes:
o- Identification of the major modules/components that will-
be separately installed items;
o Identification .of the facilities and location(s) at
which the system and data base will be installed, and
the specific modules/components to be installed at each
location;
o Identification for each module/component installed at
each location:
Installation date,
- Special conditions (if any),
Organizations and specific personnel to perform the
installation, and
-- Organizations and specific personnel on call to
. support the installation;
o Mechanisms to ensure effective software integration and
data bases synchronization for system modules/components
installed at multiple locations.
2.16. User Support Approach
This section draws from the System Concept, Detailed
Functional Requirements, and System Design to provide a summary
of the activities and materials to be used to conduct initial
system training and provide ongoing user support. It is included
in the Project Management Plan to provide an overview of the user
support approach needed to prepare and review the Project
Workplan and other sections of the Project Management Plan.
16
-------
OSWER DIRECTIVE #9Q2B.03a
Specific information presented in this section at a summary level
includes:
o Lead organizations for planning and conducting training
and ongoing user support;
o Identification of individual training sessions to be
conducted in support of initial system implementation,
and for each session:
Location, date and time,
Intended trainees and subject material (e.g., data
entry/edit/update procedures, reporting and
retrieval, system administration, etc.)r
«
Session format (group presentation/demonstration,
one-on-one training), and
Organizations and individuals who will conduct
training;
o Identification of other training activities/materials,
such as tutorials, computer-based training, etc.
o Identification of user support functions such as
hotlines, user groups, etc.., including for each
function:
Function identification,
Expected duration,
Staffing level,
Assignments of specific organizations and
individuals, and
Physical location(s).
2.17. Maintenance Approach
This section draws from the System Concept, Detailed
Functional Requirements, and System Design to provide a summary
of the organizational approach for maintaining the system.
System maintenance is crucial to the ongoing viability of the
system. For distributed systems, system maintenance is
particularly challenging, and the Maintenance Approach takes on
added importance. This section is included in the Project
Management Plan to provide an overview of the Maintenance
Approach needed to prepare and review the Project Workplan and
other sections of the Project Management Plan. Specific
information presented in this section includes:
o Identification of organizations responsible for
performing software maintenance for each system module;
17
-------
33 DP.ECTIVE ssrsa.oo
o Identification of organizations responsible for
maintaining applications software packages, including
any customized portions of the package;
o Identification of organizations responsible for
maintaining each interface with other automated systems
and data bases; .
o Identification of organizations responsible for
supporting the release of new software (routine
maintenance and enhancements) at each location where the
" system is installed; and
o Identification of currently planned maintenance and
system enhancement releases, and the content and
installation schedule for each release.
Other documents provide additional, detailed information
regarding system maintenance: details of software libraries and
maintenance procedures are documented in the Maintenance Manual,
and are summarized in the Configuration Management Plan (another
section of the Project Management Plan). These documents should
be specifically referenced by the Maintenance Plan.
2.18. Operations Approach
This section draws from the System Concept, Detailed
Functional Requirements, and System Design to provide a summary
of the organizational approach for operating the system. It is
included in the Project Management Plan to help ensure that all
organizations with system operations responsibilities are clearly
designated and are informed of their responsibilities. Specific
information presented in this section includes:
o Identification of organizations responsible for
performing basic system operations data entry,
update, and reporting for each module of the system;
Identification of organizations responsible for
performing system and data base backup and recovery for
each facility (including individual microcomputers)
where the system is installed;
each facility (including i
where the system is installed;
o Identification of organizations responsible for
performing other system administration functions (e.g.,
maintenance of data tables) for each facility where the
system is installed; and
o Reference to the Data Management Plan for the system to
identify organizations responsible for other data
administration functions.
Other documents provide additional, detailed information
regarding system operations: details of operating procedures are
18
-------
documented in the Operation Manual and User Manual. Organiza-
tions responsible for providing technical support to users are
identified in the User Support Plan (another section of the
Project Management Plan). These documents should be specifically
referenced .by the Operations Plan.
3. DEVELOPMENT AND UPDATE OF THE PROJECT MANAGEMENT PLAN
3.1. Responsibility for Developing and Updating the Project
Management Plan
The Project Manager is responsible for developing the
Project Management Plan and for keeping it current throughout the
system life cycle. An out of date Project Management Plan is not
useful to guide the project, and could lead to confusion among
project participants. The Project Manager may be assisted by
other individuals as appropriate.
3.2. Format of the Project Management Plan
All sections of the Project Management Plan should be kept
in a single document, organized in accordance with the major.
topics of the Plan. . Use of a three-ring binder or .binders is
recommended. For those portions of the Project Management Plan
developed and maintained using .automated project management
tools, current outputs of the tools should be included in the
binder, if possible. Certain sensitive sections of the Plan that
should not be readily available to all team members, such as the
details of the procurement approach, may be maintained in a
separate binder.
3.3. Evolution of the Project Management Plan Through the System
Life Cycle
The Project Management Plan evolves over the course of the
system life cycle, including a subset of topics at the end of the
Initiation phase, and evolving into a comprehensive plan by the
end of the Concept phase. Exhibit 3-1 illustrates'the evolution
of the Project Management Plan through the system life cycle.
At the end of the Initiation phase, the plan should contain
information about several topics, as shown in Exhibit 3-1. At
this time, only limited information is known about the
information management problem or the potential solutions. Thus,
the Project Management Plan, contains some basic information about
the project, and a workplan for the Concept phase. Specific
topics addressed in this first draft of the Project Management
Plan are:
o Project Charter/Objectives - Includes identification of
the information management -problem to be solved, the
pertinent programmatic mission, and a preliminary view
19
-------
OSWER DIRECTIVE *322a.03a
EXHIBIT 3-1: EVOLUTION OF PROJECT MANAGEMENT PLAN
THROUGH THE SYSTEM LIFE CYCLE
PHASE/STAGE
TOPIC
cu
o
te.
Project Charter/Objectives
Life Cycle Adjustment
Project Team Organization
Project Budget
Project Reviews/Quality Assurance
Applicable Project Approvals
Benefit-Cost Analysis
Methodologies and Tools
Workplan
Procurement Approach
Configuration Management Plan
Documentation Standards
Security Approach
Conversion Approach
Installation Approach
User Support Approach
Maintenance Approach
Operations Approach
LEGEND;
ART mREFINE AS NEEDED
OMPLETE m EXPAND ANDlOR ADD DETAIL WEW ITERATION COMPLETE
20
-------
QSWffl DIRECTIVE ^3C2
of the scope of the problem in terms of the functions
and organizations experiencing the problem. This
section also identifies the project sponsor.
o Life Cycle Adjustment - Includes any adjustments to the
life cycle to be made in the Concept phase. For
example, for a relatively simple problem, the more
detailed functional and data requirements normally
performed during the Definition stage might be included
in the Concept phase.
o Project Team Organization - Includes identification of
the Project Manager, and the project participants and
project team structure for the Concept phase. This
section identifies participating organizations and
individuals for the Concept phase, and also identifies
the intended use of contractor support.
o Project Budget - Identifies the total resources needed
to conduct the Concept phase', and includes a preliminary
order of magnitude preliminary estimate of the aggregate
cost of all other stages through the Implementation
stage (e.g., whether the aggregate cost should be viewed
in terms of thousands, hundreds of thousands, or.
millions of dollars, and commensurate EPA workyears).
o Project Reviews/Quality Assurance - Identifies the
preliminary threshold level for the project based on the
known information about the problem. {How the
'threshold analysis' should be conducted is described in
the practice paper for 'System Life Cycle Reviews and
Approvals'.) Also identifies the lead reviewer for the
project and a scheduled dates for completion of the
Initiation phase and Concept phase reviews.
o Applicable Project Approvals - Based on the preliminary
threshold level for the project and known information
about the problem, identifies the approval authority (in
terms of specific organization(s) and individual(s)) for
the Initiation and Concept phases of the project.
o Benefit-Cost Analysis - Provides only a rough order of
magnitude estimate of the project costs (based on the
budget estimates described above) and a brief narrative
statement of the expected benefits and the organizations
that will realize them. A quantitative estimate of
benefits is not essential in the Initiation phase.
o Methodologies and Tools - Identifies the analytic
methods and automated tools .that will be used in the
Concept phase.
o Workplan - Describes the tasks to be conducted in the
Concept stage, noting for each task who will perform the
21
-------
OSWER DIRECTIVE #9028.008
task, its schedule, resources to be used, and products
to be generated. The Workplan should include a summary
prepared in a Gantt chart format whenever possible. The
Workplan also identifies at this time any key
assumptions or constraints on conducting the tasks of
the Concept stage.
*"'
>'
The Concept phase adds considerable new information to the
Project Management Plan, introducing most of the remaining topics
and adding detail to the topics first addressed during the
Initiation phase. By the end of this phase, the Project
Management Plan is comprehensive. The Concept phase results in
the selection of a specific alternative for solving the
information management problem, and the Project Management Plan
describes the management approach for taking that alternative
through the remainder of the system life cycle.
In succeeding phases and stages, the Project Management
Plan is updated and refined as necessary, based on the results of
project activities. Some sections of1 the Project Management Plan
may change significantly to address difficulties encountered in
managing the project. Any changes to the basic system concept
will likely result in changes to one or more elements of the
Project Management Plan, with the Workplan and Project Budget the.
most likely to change. If at any time it becomes necessary to
significantly change any part of the Project Management Plan, the
Project Manager should retain the prior version for reference
purposes and to support the .post-implementation evaluation of the
system. .
3.4 Retention of Old Project Management Plans
The Project Management Plan-is a living document, evolving
continuously throughout the system life cycle. Although keeping
the Project Management Plan current is important, it is also
important to preserve prior versions of the plan to preserve a
record of the evolution of the project. This record will be very
useful if there is a changeover in project management it will
enable the new Project Manager to more easily 'come up to speed'
on both- the current status of the project and its history. 'In
addition, this record will be extremely useful if the project is
audited by the Agency's Office of the Inspector General (OIG).
It will enable the project team, and the project sponsor, to
provide the information requested by the OIG much more easily and
ensure that the information provided is accurate.
Although the Project Management Plan may be refined
relatively frequently, the Guidance does not require the same
extent of recordkeeping as for other system documents, those
contained in system baselines. To ensure that a complete record
of the Project Management Plan history is retained, a copy of the
current Project Management Plan should be filed with each
approved System Decision Paper, in the same baseline as the
corresponding System Decision Paper. For most projects, this
22
-------
QSWER DIRECTIVE £SQ£3.CCa
procedure will result in filing a copy of the Project Management
Plan at the end of each phase and stage of the system life cycle.
4. RELATIONSHIPS BETWEEN PROJECT MANAGEMENT PLAN TOPICS
The topics of the Project Management Plan are related to
each other in that they address different perspectives of the
same project management issues. It is therefore important that
the contents of the Project Management Plan be internally
consistent. For example, the cost of contractor resources to
conduct the activities enumerated in the project Workplan must be
consistent with available contract funding identified in the
Project Budget. Similarly,' the intended use of contractor
support must be reflected in the Procurement Approach to ensure
that the means for acquiring such support (e.g., signed
contracts) are in place in a timely manner. Exhibit 4-1
identifies all significant relationships among the topics of the
Project Management Plan. Exhibit 4-2 describes each of these
relationships.
23
-------
OSWES DIREC
EXHIBIT 4-1: SUMMARY OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT PLAN TOPICS
Li
Project
Ass
ie
is
Cost
S
Appro
Approac
User S
Mainte
1
a
Project Charter/Objectives
Life Cycle Adjustment
Project Team Organization
Project Budget
Project Reviews/Quality Assurance
Applicable Project Approvals
Benefit-Cost Analysis
Methodologies and Tools
Workplan
Procurement Approach
Configuration Management Plan
Documentation Standards
Security Approach
Conversion Approach
Installation Approach
User Support Approach
Maintenance Approach
Operations Approach
Designates two topics that address one or more common subjects,
and that should treat these subjects in a consistent manner. For
example, the use of contractors as shown in a Project Workplan
should be reflected as well in the Procurement Plan.
24
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Project
Charter/
Objectives
Related Topic Relationship
Kl
in
Life Cycle
Adjustments
Project Team
Organization
Project
Reviews/
Quality
Assurance
Applicable
Project
Approvals
Project Team
Organization
Project Budget
Project
Reviews/
Quality
Assurance
The lead organization for the project is
designated in the Project Charter.
The Project Charter designates the threshhold
level for the system, and one or more of the
organizations that should participate in project
reviews.
The Project Charter designates the threshhold
level for the system, and one or more of the
organizations that should participate in project
reviews.
The project team structure and/or participants
may change during the life cycle, and should
reflect any life cycle adjustments. For large
systems, adjustments to the life cycle that
provide a phased development and implementation
of major modules or work packages should be
reflected in the project team organization.
The project budget, which is usually presented to
follow the standard phases and stages, should be
structured to reflect any adjustments to the life
cycle.
Life cycle adjustments which serve to combine
parts of phases and stages may affect the number
and/or timing or project reviews. The specific
reviews to be conducted should reflect any such
adjustments.
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Life Cycle
Adjustments
(continued)
Related Topic Relationship
to
o»
Applicable
Project
Approvals
Methodologies
and Tools
Workplan
Procurement
Approach
Configuration
Management Plan
Conversion
Approach
Life cycle adjustments which serve to combine
parts of phases and stages may affect the number
and/or timing or project approvals. The specific
approvals to be conducted should reflect any such
adjustments.
The identification of specific methodologies and
tools should be consistent with any adjustments
to the life cycle. Also, some tools, such as
program code generators, by themselves define an
adjustment to the life cycle in that activities
conducted in one stage (e.g., Design) may
generate automatically products that are
typically produced in another stage (e.g.,
Development).
The project Workplan must be consistent with any
adjustments to the phases and stages of the life
cycle. Each task/activity should be associated
with a specific phase or stage.
The Procurement Approach identifies the resources
to be acquired for each phase and stage, and
should be consistent with any life cycle
adjustments.
Adjustments to the life cycle should be reflected
in the products associated with each baseline.
The content of some baselines may differ from the
typical set products.
Life cycle adjustments, particularly those that
provide a phased development of modules or work
packages, may require a comparable phasing of
data conversion activities for each module.
S
p
I
m
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Life Cycle
Adjustments
(continued)
Project Team
Organization
KI
Related Topic
Installation
Approach
Project
Reviews/...
Quality
Assurance
Applicable
Project
Approvals
Workplan
Procurement
Approach
Relationship
Life cycle adjustments, particularly those that
provide a phased development of modules or work
packages, may require a comparable phasing of
installation activities for each module.
The Project Team Organization should identify the
organizations, and individuals from each
organization, that will perform formal project
reviews and other quality assurance activities.
The designated organizations should be consistent
with the results of the 'threshold analysis'
determining the appropriate level of project
review and approval.
The Project Team Organization should identify the
organizations, and individuals from each
organization, that will provide required project
approvals. The designated organizations should
be consistent with the results of the 'threshold
analysis' determining the appropriate level of
project review and approval.
The assignment of specific tasks presented in the
project Workplan should reflect the specific
organizations and individuals identified in the .
Project Team Organization.
The use of contractors to conduct the project is
identified in the Project Team Organization.
Each contract vehicle under which contractor
support is to be obtained, including new
procurements, should be identified in the
Procurement Approach
g
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Project Team
Organization
Related Topic Relationship
Configuration
Management Plan
Conversion
Approach
N>
oo
Installation
Approach
User Support
Approach
Maintenance
Approach
The Project Team Organization should identify
specific organizations and 'individuals who will
perform specific configuration management
functions identified in the Configuration
Management Plan: Configuration Manager, and
Change Control Panel.
The Project Team Organization should reflect the
content of the Conversion Approach regarding the
specific organizations responsible for
determining the data to be converted to the new
system/data base, and for accomplishing the data
conversion.
The Project Team Organization should reflect the
content of the Installation Approach regarding
the specific organizations responsible for
supporting system installation, including
installations at all locations for distributed
systems.
The Project Team Organization should reflect the
content of the User Support Approach regarding
the specific organizations responsible for
providing initial system training and performing
ongoing user support activities.
The Project Team Organization should reflect the
content of the Maintenance Approach regarding the
specific organizations responsible for performing
maintenance activities, including maintenance of
interfaces with other systems and data bases.
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Related Topic Relationship
Project Team
Organization
Project Budget
K>
Project
Reviews/
Quality
Assurance
Operations
Approach
Benefit-Cost
Analysis
Workplan
Procurement
Approach
Applicable
Project
Approvals
Benefit-Cost
Analysis
Workplan
The Project Team Organization should reflect the
content of the Operations Approach regarding the
speqific organizations responsible for operations
support, including support of operations at each
location for distributed systems.
The costs documented in the Project Budget should
be consistent with the costs identified in the
Benefit-Cost Analysis.
The resources identified by task in the Workplan
should be, in the aggregate, consistent with the
resources provided in the Project Budget for each
type of resource.
All contract resources identified in the Project
Budget should also be reflected in the
Procurement Approach '
The level of organization(s) responsible for
conducting project reviews should be consistent
with the results of the threshold analysis for
determining the appropriate level of project
approvals.
Benefit-Cost Analyses that show project costs
(one time investment, annual costs, and/or total
life cycle) greater than OMB and/or OIRM
reporting requirements require formal project
reviews conducted by the SIRMO and appropriate
Information Management Coordinators.
Individual project reviews and other quality
assurance activities should be specifically
identified and scheduled in the project Workplan.
1
SO
a
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Project
Reviews/
Quality
Assurance
(continued)
Applicable
Project
Approvals
Related Topic Relationship
u>
o
Methodologies
and Tools
Configuration
Management Plan
Benefit-Cost
Analysis
Workplan
Workplan
Procurement
Approach
Configuration
Management Plan
Documentation
Standards
Security
Approach
Specific organizations and individuals who will
participate in reviews of requested changes to
the system are designated as the Change Control
Panel and are identified under both topics.
Benefit-Cost Analyses that show project costs
(one time investment, annual costs, and/or total
life cycle) greater than OMB and/or OIRM
reporting requirements require project approvals
by the Information Management Steering Committee.
Individual project approvals should be
specifically identified and scheduled in the
project Workplan.
Tasks for selecting project methodologies and
tools should be specifically identified and
scheduled in the project Workplan.
The acquisition of automated tools not currently
owned or leased by the Agency should be included
in.the Procurement Approach
The identification of automated tools, if any, to
support configuration management activities
should be consistent under both of these topics.
The selection of methodologies and tools should
be consistent with, and support, the selected
documentation standards.
The identification of automated tools, if any, to
design, implement or assess system and data base
security should be consistent under both of these
topics.
-------
EXHIBIT 4-2i DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Methodologies
and Tools
(continued)
Related Topic Relationship
Workplan
Conversion
Approach
Installation
Approach
Maintenance
Approach
Procurement
Approach
Configuration
Management Plan
Security
Approach
Conversion
Approach
The identification of automated tools, if any, to
support data conversion should be consistent
under both of these topics.
The identification of automated tools, if any, to
support system installation should be consistent
under both of these topics.
The identification of automated tools, if any, to
support software maintenance should be consistent
under both of these topics.
Activities for accomplishing the acquisitions
identified in the Procurement Approach should be
specifically included and scheduled in the
project Workplan.
Activities supporting 'the development of the
Configuration Management Plan, and its
implementation, should be specifically included
and scheduled in the project Workplan.
Scheduling of meetings of the Change Control
Panel should be included in the Workplan.
Activities supporting the development of the
Security Approach and its implementation, should
be specifically included and scheduled in the
project Workplan.
Activities supporting the development of the
Conversion Approach should be specifically
included and scheduled in the project Workplan.
Activities for accomplishing the conversion of
data into the new system or data base also should
be included and scheduled in the Workplan.
-------
EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Workplan
(continued)
Related Topic Relationship
Ul
K)
Procurement
Approach
Installation
Approach
User Support
Approach
Maintenance
Approach
Operation
Approach
Security
Approach
Conversion
Approach
Installation
Approach
Activities supporting the development of the
Installation Approach, and its implementation,
should be specifically included and scheduled in
the project Workplan.
Activities supporting the development of the User
Support Approach, and its implementation, should
be specifically included and scheduled in the
project Workplan.
Activities supporting the development of the
Maintenance Approach, and its implementation,
should be specifically included and scheduled in
the project Workplan.
Activities supporting the development of the
Operation Approach, and its implementation,
should be specifically included and scheduled in
the'project Workplan.
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
provide system or data base security should be
included in the Procurement Approach.
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
support data conversion, or the use 'of contractor
support to accomplish data conversion, should be
included in the Procurement Approach.
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
support system installation, or the use of
contractor support to install the system, should
be ineeded in the Procurement Approach.
m
s
ll'
«
I
-------
EXHIBIT 4-2: DETAILS OP RELATIONSHIPS
BETWEEN PROJECT MANAGEMENT TOPICS
Topic
Procurement
Approach
(continued)
Related Topic
User Support
Approach
Configuration
Management Plan
. Maintenance
Approach
Operation
Approach
Maintenance
Approach
Conversion
Approach
Operations
Approach
Installation
Approach
Relationship
*
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
conduct training and other user support
activities, or the use of contractors to provide
user support, should be included in the
Procurement Approach.
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
perform system maintenance, or the use of
contractors to maintain the system, should be
included in the Procurement Approach.
The use of contractors to operate the system
should be included in the Procurement Approach.
Procedures for assessing requested changes to the
system, including maintenance changes, and for
tracking the status of change requests, are
included in the Configuration Management Plan.
The organizations responsible for maintaining the
system should be represented on the Change
Control Panel designated in the Configuration
Management Plan.
Procedures used to record anomalies of system
operations, and to track their resolution, are
identified in the Configuration Management Plan.
Installation of the system should generally
provide for the loading of existing data prior to
cutover to full production. The Installation
Approach should refer to the Conversion Approach
for the details of accomplishing data conversion
activities.
-------
APPENDIX A
DETAILED OUTLINE OF PROJECT MANAGEMENT PLAN
34
-------
PROJECT MANAGEMENT PLAN
SUMMARY DESCRIPTION
The Project Management Plan is created during the Initiation phase and updated in
each phase or stage of the system life cycle. Some topics (e.g., security
approach, maintenance approach) are summarized in the Project Management Plan,
and presented in greater detail in other life cycle products.
TOPICS
to
in
o Project charter/objectives
Project identification (incorporate
Initiation Decision Paper by reference)
Mission and objectives
Scope of information management
problem/project
o Life cycle adjustment
Consolidation of phases and stages, if
any
Partitioning of project/system into
major work packages, modules, etc. with
different timing through the life cycle
o Project team organization
Project management structure
Manager assigned: individual,
current organization, authority
Boards, committees, or other
project management participants
Changes or additions for Operation
phase
Project team organization
Structure and roles
Participating organizations
Staffing plan (including internal
staff and use of contractors)
Changes or additions for Operation
phase
Other organizations to be notified of
major project events (non-participants
in project team)
I
*
co
-------
PROJECT MANAGEMENT PLAN (Continued)
to
o Project budget (broken out by stage)
~ EPA staff
Contractor services
Equipment acquisition
Hardware maintenance
Site preparation
Packaged software acquisition
Supplies
Timeshare
~ Other
Cost-accounting methodology
o Project reviews/quality assurance
Applicable project review level
Reviews to be conducted (by stage)
Organization/individuals for each
review
Review schedule
o Applicable project approvals
Project approval level
Specific approvals to be obtained (by
stage)
Approval organization and individuals
Approval schedule
o Benefit-cost analysis (summary, transferred
from other life cycle products)
Methodology and assumptions
-- Benefits
Programmatic
Annual monetary
System life
~ Costs
Nonrecurring
Recurring
Annual
System life
Payback period
Sensitivity analysis
o Methodologies and tools
~ Methodologies (non-automated)
For Concept phase
For Definition stage
For Design stage
For Development stage
For Implementation stage
For Production stage
For Evaluation stage
For Archive stage
o
-------
PROJECT MANAGEMENT PLAN (Continued)
Automated tools/software packages
For Concept phase
For Definition stage
For Development stage
For Implementation stage
For Production stage
For Evaluation stage
For Archive stage
Support required (if any) for use
of tools
o Workplan
U) Concept Phase
Definition Stage
Development stage
Implementation stage
Production stage
Evaluation stage
Archive stage
Activities and related tasks
Products
Schedule by task and product
Staff and contractor assignments
Level of resources for each task
and/or product
Task relationships/dependencies
Schedule of reviews' and approval
Performance/progress reporting
. - Notification
o Procurement approach *
Resources to be acquired through
existing contracts
OSNER contracts
Other agency contracts
Resources to be acquired through new
procurements
- OSWER vehicles
Other Agency vehicles
Schedule for each procurement
Norkplan for each OSWER procurement
Procurement assistance individuals
for each procurement
o Configuration Management Plan
Configuration manager (organization and
individual)
Change Control Board
Participants (organizations and
individuals)
Modification request/approval
process
Procedures/methods for configuration
identification and accounting, software
control, audits
Configuration management documentation:
identification and location of existing
CM logs, and official existing baseline
contents
-------
PROJECT MANAGEMENT PLAN (Continued)
00
o Documentation standards: Standards to be
used for each life cycle product
o Security approach
Summary of security requirements
(reference other life cycle products)
Security organization (it applicable)
Hardware and facilities measures
Software and communications measures
Data base security
Procedural measures
o Conversion approach
Overview .
Data identification
Current data location
Organizations to accomplish '
conversion
Manual data to be converted
Sources
Procedures N
Error conditions to be corrected
Automated data to be converted
Sources
Procedures
Error conditions to be corrected
o Installation approach: Schedule for
installing each separately-installed system
module
Dates and times, by module and location
Special conditions
Personnel to accomplish installation,
and/or on call
o User support approach
Training activities
Materials to be prepared
Sessions, schedules, and
participants
Ongoing user support (hotline, etc.)
o Maintenance approach
Maintenance support organization
Release management procedures
Planned maintenance releases
o Operation approach
: Organization of operation support
activities
Reference to Operation Manual
-------
OSWER DIRECTIVE #9328.003
OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
(OSWER)
SYSTEM LIFE CYCLE
MANAGEMENT GUIDANCE
Part 3: Practice Paper
Configuration Management
January, 1989
-------
OSWER DIRECTIVE *302G.Kto
TABLE OF CONTENTS
1. Practice Paper Purpose 1
2. Overview of Configuration Management Activities and
Practices 2
«
2.1. Configuration Management Overview 2
2.2. Configuration Item Identification 3
2.3. System Baseline Management 5
2.4. Change Control 6
2.5. Software Control 9
2.6. Configuration Accounting 9
2.7. Configuration Audits 10
3. 'Configuration Management Organization 10
3.1. Configuration Manager 11
3.2. Change Control Board .' 11
4. Configuration Management Plan ' 13
4.1. Purpose of Configuration Management Plan 13
4.2. Development and Update of the Configuration
Management Plan 13
4.3. Description of Configuration Management
Plan Topics 13
s
EXHIBITS
2-1. Configuration Management Through the System
Life Cycle 4
2-2. Contents of System Baselines 7
4-1. Evolution of Configuration Management Plan
Through the System Life Cycle 14
4-2. Outline of Configuration Management Plan 15
-------
OSWER DIRECTIVE 3SOSE.C03
1. PRACTICE PAPER PURPOSE
This practice paper constitutes a section of Part 3 of the
Office of Solid Waste and Emergency Response (OSWER) System Life
Cycle Management Guidance. It describes OSWER's practice of
configuration management, tailoring industry proven configuration
management methods and techniques to the OSWER environment.
Configuration' Management is a function which serves to
systematically identify the characteristics of a system (referred
to as "configuration items"), and formally control any changes or
additions to those items. Examples of configuration items
include specific functional and data requirements, the system
design, and documentation such as the User Manual. Configuration
Management helps maintain the integrity of the system throughout
its life cycle, and facilitates communication about the system
among system project team members, users, and other supporting
organizations.
This practice paper provides the following guidance
regarding the implementation of configuration management:
o Describes specific activities associated with.
configuration management;
o Describes project organization structures to accomplish
configuration management; and
o Describes the .documentation of project-specific
configuration management activities in a Configuration
Management Plan.
Ensuring the integrity of the system may be particularly
challenging for a large and/or complex system. A rigorous and
disciplined configuration management function will maintain
system integrity for these systems, and all other systems as
well, in the following ways:
o .Helps ensure clear identification and documentation of
functional and data requirements;
o Helps ensure that all approved requirements are
traceable through the system design;
o Provides a means to unambiguously identify and reference
the specific components of the system design and the
actual system (e.g., subsystems, data bases,
documentation);
o Helps ensure that an adequate review of requested
changes to the system, and approval by an authorized
organization and individual, takes place before the
system is changed;
-------
o Helps ensure effective control over changes to the
software and release of changes to users;
o Provides a means to clearly identify the status of the
system, and of individual system components, at any time
throughout the life cycle;
o Helps ensure the consistency of products, such as the
agreement of documentation with the applications
software and other components of the system;
o Facilitates effective communication among system project
team members, users, and other supporting organizations
about the characteristics of the system and its status
throughout the system life cycle; and
o Provides a means to reconstruct the evolution of the
system, and rationale for significant decisions, through
the prior phases and stages of the system life cycle.
This practice paper is not intended to provide detailed
configuration management procedures. Rather, it describes the
activities, organizational framework, and types of procedures
that should be adopted by each system project for effective.
configuration management. The details of the procedures for a
given project should be developed on a case by case basis to
reflect the program needs, technical environment, and involved
organizations for each project. ' '
2. OVERVIEW OF CONFIGURATION MANAGEMENT
ACTIVITIES AND PRACTICES
2.1. Configuration Management Overview
Configuration 'management encompasses six sets of related
activities:
t
o Configuration Item Identification which identifies the
characteristics of the system to be controlled by
delineating specific configuration items;
o Baseline Management, which establishes repositories (or
libraries) that contain the documentation of these
items;
o Change Control, which ensures that only, advantageous
changes are made to the system as it exists in the
controlled baselines;
-------
OSWcR DIRECTIVE iSO'i5.Sr:3
o Software Control/ which preserves the integrity of the
software while changes are being made;
o Configuration Accounting/ which monitors the status of
theconfigurationitems, and requested changes to the
system in terms of their impact on specific
configuration items; and
o Configuration Audits/ which confirm the proper working
of the configuration management function and help ensure
that system documentation is complete and current.
These activities are performed throughout the system life
cycle/ starting with the Initiation phase and continuing through
the end of system operations in the Archive stage. Exhibit 2-1
provides an overview of configuration management throughout the
system life cycle.
2.2. Configuration Item Identification
Configuration item identification serves to clearly
delineate the significant characteristics of the system/
providing a common language/ or referencing scheme/ for.
describing the system. It is the delineation of specific
configuration items that will enable all individuals involved in
the evolution of the system to communicate effectively/ using the
common language. Examples of such characteristics include
specific functional and data requirements/ specific
characteristics of the system design (e.g./ subsystems/ files/
(etc.) and system documentation. Each characteristic, or set of
related characteristics/ is- referred to as a configuration item.
The delineation of individual configuration items can be. viewed
as creating the 'labels' for the characteristics described in the
conventional documentation of the system. For example, the
design for a system's application programs that produce reports
for use by regional offices may be delineated collectively as a
single configuration item called 'Regional Report Programs'.
When . conducting configuration item identification/ be sure
to note the following practices:
o Configuration items delineate different types of system
characteristics, and reflect the current phase and stage
of the life cycle.
Individual functional and data requirements/ or sets
of related requirements, are delineated in the
Definition stage;
Major attributes of the system design, are
delineated in the Design stage, such as:
-------
EXHIBIT 2-1: CONFIGURATION MANAGEMENT
THROUGH THE SYSTEM LIFE CYCLE
Initiation Baseline
LOO/RESPOND TO CHANGES
DeTmitioo Btulioe
Deiip Bateline
Development Baseline
Definition Baseline
|iprf.|ftd-
Initiation Baseline
Iniliatioo Bauline
Definition Bucline
Devdofnieni Boeline
Opcfational Bucline
r- i
Inilitlioo Baieline
Definition Baseline
Design Baseline
Development Baseli
Updated:
Initiation Baseline
Definition Baseline
Design Baseline
-------
OSWER DIRECTIVE ^u^.
- Hardware and technical environment(s)
- Logical data base structures
- Physical data base structures
- Applications software
- Utility software
The application software (i.e., programs), such as
input and update processing, reporting, and system
administration processes are delineated in the
Design stage.
Specific system documentation, such as User Manual,
Maintenance Manual, and Operations Manual are
delineated in the Development stage.
o Configuration items are delineated and recorded in the
normal documentation of the system life cycle. An index
of configuration items should be included in each
documentation product.
The index identifies specific configuration items.
The index also identifies specific relationships
between configuration items in the same product.
(e.g., between different parts of the system
design), and also relationships with previously
defined items (e.g., between design configuration
items and requirements configuration items).
The index helps confirm the traceability of the
system throughout the life cycle, and is a vital
tool to accomplish system audits.
2.3. System Baseline Management
System baselines are collections of life cycle products,
including hardware and software, as well as the documentation of
the system. An easy way to understand what a baseline is is to
view it as a controlled library collection. Creating and
managing, .baselines does not require the development of
significant additional system documentation. System baselines
are established throughout the life cycle to establish a clear
basis for monitoring the status and progress of the system, and
to facilitate communication (particularly for large system
projects).
Five baselines are developed, and maintained, throughout
the life cycle:
o Initiation Baseline - Contains documentation of the
information management problem,
-------
OSWER DIRECTIVE *S023.03$
o Definition Baseline - Contains documentation of the
functional and data requirements,
o Design Baseline - Contains documentation of all
attributes of the system design/
o Development Baseline - Contains the automated and
hardcopy products of development (including
documentation), and of system changes, prior to
installation in the user environment, and
*
o Operational Baseline - contains the automated and
hardcopy products for the currently installed system.
Each baseline contains different life cycle products, and
provides a different perspective of the system. Exhibit 2-2
illustrates the content of each baseline. Important
characteristics of baselines and their development include the
following:
o Baselines contain approved products only. Project teams
should use other libraries to monitor the status of and
control products that are under development and not yet
approved.
o Baselines contain the product as first approved, and
subsequent approved changes. For documentation, changes
may be added to the baseline in the form of an entirely
new document, a replacement document, or as an addenda
or replacement pages.
o All baselines are maintained throughout the life of the
system, not just the operational system. If the
information management problem changes, or the approved
requirements for the system or system design change, it
is essential to keep all of the established baselines
current in order to have available complete, accurate,
documented information about all aspects of the system.
2.4. Change Control
Change control is a formal process for determining what
changes are to be made to the system. Change control requires
the documentation of specific requests for modifications, and a
review of the requested modifications, and consideration of their
impact on the system, before they are made. The term
'modifications' refers both to changes and to minor enhancements.
Change control determines which requested modifications will be
made and which will not be made. .
Change control applies to modifications requested after the
system concept is approved and before the system is implemented,
as well as to requests to modify fully operational system. For a
system under development, the process for approving changes
-------
EXHIBIT 2-2: CONTENTST)F SYSTEM BASELINES
/System Concept \
Initiation Dccliion
Paper \
ZAcctfUoc* Tea Dncumeni
, _ .
/ Sygem Tea
A)K» Support Maieiiali (Dev.V
/Security Manua
/Oftiuiont Mtnu.l
KUinlcnmce Manual (Dev.)V
v»
liter Manual (DevelopmeniA
/Uicr Support V
Matcriali (Prod) \
/Security Manual (OpcXProd )\
/OpctaUooi Manual (l*rod)~X
/MainlCTance Manual (Prod.) \"
Uier Manual (Production) V
ill
S';
III
I)
IVI
-------
OSWER DIRECTIVE *SC23.COa
may be integrated with the formal approval process that takes
place at the end of each stage of the life cycle. However, since
these formal approvals are occur at the end of the stage, it is
better to establish a separate change control process, and
include a summary of changes approved using this process as part
of the System Decision Paper submitted for formal approval at the
end of each stage.
Other important aspects of change control include the
following:
o Change control provides an examination of requested
changes to the system before the system is changed.
This examination is referred to as a 'Change Request
Impact Analysis'. This analysis considers all pertinent
characteristics of the change:
Functional and/or data requirement,
;
Reason for the change (e.g., new regulation or
program policy),
Benefits of making the change,
Impact of not making the change on OSWER program
operations, and .the associated organizations,
Specific system configuration items impacted by the
change, and extent of impact,
Impact of the change on related systems and data
bases,
Cost of accomplishing the change, (including both
onetime costs, and recurring costs [operations and
maintenance], and expected source of funds),
Impact of the change on user organization procedures
and/or staffing requirements,
Timeframe for accomplishing the change,
Potential risks in making the change (i.e., is
successful change a certainty?),
Disposition of prior requests for the same change.or
a comparable change.
o Requested changes for a system under development may
apply to system characteristics that have not yet been
approved and recorded in a baseline. The Project
Manager should design a process for determining how to
handle these requests, and document the process in the
8
-------
OSWER DIRECTIVE #3023.00:
Configuration Management Plan. The process may be the
same one used to decide how to handle requested changes
to previously approved configuration items.
2.5. Software Control
Software control is a set of procedures to ensure that the
integrity of the system is preserved when approved changes are
being made to the system, or in the event of a disaster and
restoration of the system is needed. Software control procedures
are particularly important during the Production stage of the
life cycle. Software control ensures that changes to the
computer programs are developed and tested using a copy of the
programs and a test data base, and do not adversely affect system
users. However, software control is also important during the
Development and Implementation stages to control changes during
the initial building and installation of the system. For each
system, the software control procedure should cover the following
points:
o Describes how the . development and operational
environments of the system will be segregated.
o Describes the procedures to be used to install new
versions of the software in the production environment,
including procedures to ensure that a system installed
at multiple locations '(e.g., regional logical
mainframes, multiple personal computers, etc.) is
properly integrated and distributed data bases are
effectively synchronized.
Other topics may be included as appropriate to the
system.
2.6. Configuration Accounting
Configuration Accounting is an administrative procedure for
maintaining system baselines and monitoring the status of the
system throughout the life cycle. Important elements of this
procedure include:
o Logging and storing each life cycle product in the
appropriate baselines upon the approval of the product.
o Recording all requested changes, maintaining a log of
change requests, and also recording the disposition of
each request (e.g., approved, held pending further
analysis, disapproved).
o Monitoring the status of approved changes, and recording
completed changes in the appropriate baseline. Specific
tracking points are determined for each system and/or
change as appropriate.
-------
o Providing the records used to accomplish configuration
audits.
The Configuration Accounting procedure may also include
activities for producing copies of approved and baselined system
documentation for use by project team members (including
contractors) throughout the life cycle»
it
2.7. ''Configuration Audits
Configuration Audits are examinations of the products and
related documents submitted for inclusion in a baseline to assure
that they are complete, clearly presented, and internally
consistent. This examination is oriented to adherence to
guidance and standards. These audits support formal reviews and
evaluations of the system by ensuring that required products and
documents are complete (e.g., meet identified standards), and
provide effective traceability to related products. (Of
particular note, audits do not evaluate qualitatively the
programmatic and/or technical content of the product. That is
done by formal reviews and other quality assurance activities.)
Audits help ensure that the resources used to conduct reviews and
evaluations are not applied to products that are not yet ready
for the review.
Audits also help confirm the proper working of the
configuration management function -.- they help ensure that
configuration accounting records are complete and current. As
illustrated in Exhibit 2-1, audits are conducted throughout the
life cycle.
In general,, one audit is conducted in each of the following
stages:
o Concept
o Definition
o Design
b Development
o Implementation
For the Production stage, audits ' are conducted
-periodically, usually annually or every other year, during the
Production stage. However, more frequent audits may be conducted
in any stage to ensure that deficiencies found by a recent audit
have been corrected.
3. CONFIGURATION MANAGEMENT ORGANIZATION
*
Implementation of effective configuration management
requires the formal designation of configuration management
responsibilities for each system project.
10
-------
3.1. Configuration Manager
Each system project will have a single Configuration
Manager. The Configuration Manager establishes and maintains the
configuration management records for the system. This individual
should be designated during the Concept phase, and should report
directly to the Project Manager. For a small system, the Project
Manager may serve as the Configuration Manager. Specific duties
of the Configuration Manager should include:
*'o Maintaining a complete file and log of all change
requests, including requests to modify system
characteristics that are not yet approved and baselined,
as well as requests to modify approved configuration
items.
o Recording the disposition of all change requests,
including approval/disapproval, and completion and
implementation of the change.
o Preparing periodic reports of configuration status, as
needed,
o Providing physical control over baselined documentation,
(e.g., retaining at least one official copy of the
documentation), and
o Providing assistance and support to the performance of
configuration audits.
The Configuration Manager is accountable for the
completeness and integrity of the configuration management
records, and should be an EPA employee. However, the
Configuration Manager may obtain staff support using other
members of the project team, including contractor support, as
appropriate.
The Configuration Manager usually does not have specific
decision making authority.
" /
3.2. Change Control Board
Each system project will establish a Change Control Board.
Change Control Boards examine requested changes to the system,
direct the change request impact analysis (an analysis conducted
by the project team) and, based on the results, determine the
changes that are to be made and those that are not to be made to
the system. Several important issues should be considered
carefully when establishing a Change Control Board:
o The Change Control Board should be established as soon
after the approval of the System Concept as is
. practical, and no later than the start of the "Design
stage. The. Board reviews requested changes to the
11
-------
concept, requirements, and design to the system during
the evolution of the system, as well as requested
changes to an operational system.
o The Change Control Board may act in an advisory capacity
to the Project Manager, or may serve as a decision
making body. The specific authority of the Board is
determined for each system, and is documented in the
Configuration Management Plan.
-o The Change Control Board is usually chaired by the
Project Manager, but may be chaired by any individual.
o The composition of the Change Control Board should
reflect those organizations directly affected by the
system (e.g., providers of data, direct users of the
software, and users of the system outputs).
Systems which affect multiple OSWER offices should
include a member of each office on the Board.
««.
Systems which affect the Regional offices and/or
state agencies should include at least regional
representatives on the Board, and should include.
State representatives if appropriate.
All Level I systems should include representatives
of OIRM and of the OSWER SIRMO. (Refer to practice
paper for Reviews and Approvals for definition of a
Level I system).
*
Specific individuals included on the Board should be
determined jointly by the Project Manager and the
appropriate program manager(s).
o The Change Control Board may operate under different
names, such as 'System Advisory Group', 'Board of
Directors', and 'Configuration Management Board'.
However, the name Change Control Board is preferred.
» .
o The size and composition of the Change Control Board may
change over the life of the system.
o Not all requested changes need be examined by the Change
Control Board. The Board may designate certain types of
simple, low impact changes that can be approved directly
; by the Project Manager.
o A project may elect a higher level of review for certain
types of requested changes.' The Change Control Board
may request that these changes be reviewed by the OSWER
Information Management Steering Committee.
12
-------
4. CONFIGURATION MANAGEMENT PLAN
4.1. Purpose of Configuration Management Plan
The Configuration Management Plan describes the
organization approach and specific procedures to be used to
implement configuration management for the system. It also
identifies the physical location of system baselines the
locations in which hardcopy documentation is stored, and the
automated libraries used to store other documentation and the
software components of the system.
It should be noted that the Configuration Management Plan
provides information that unauthorized individuals might try to
use to tamper with the applications software. For security
reasons, the identification of automated libraries may be stored
external to the Configuration Management Plan, in a document that
is less accessible, in order to avoid disclosing such information
to unauthorized individuals.
4.2. Development and Update of the Configuration Management Plan
The Configuration Management Plan is a component of the
Project Management Plan. It may be included in the Project
Management Plan document, or may be developed and maintained as a
separate document. The configuration accounting documents should
generally be maintained separate from the Project Management
Plan, but may be combined for very small, simple systems for
which few changes are anticipated.
The Project Manager and Configuration Manager have lead
responsibility for development of the Configuration Management
Plan.
The Configuration Management Plan is initiated early in the
life cycle, and all sections are completed by the end of the
Design stage. However, the Configuration Management Plan is
continually refined and updated throughout the system life cycle
as needed.
Exhibit 4-1 provides an outline of the Configuration
Management Plan and illustrates its evolution throughout the life
cycle.
4.3. Description of Configuration Management Plan Topics
The Configuration Management Plan should follow the general
outline and cover the topics presented in Exhibit 4-2.
13
-------
EXHIBIT 4-1:
EVOLUTION OF CONFIGURATION MANAGEMENT PLAN
THROUGH THE SYSTEM LIFE CYCLE
PHASE/STAGE
TOPIC
Introduction
Configuration Management Organization
Configuration Item Identification
Change Control Process
Configuration Accounting
Configuration Audits
v
System Baselines
Software Control
I.EfiEND;
EXPAND AND/OR ADD DETAIL
REFINE AS NEEDED
il
-------
G3WEF; D!F.2mVE FS22
EXHIBIT 4-2
OUTLINE OF CONFIGURATION MANAGEMENT PLAN
o Introduction
Identification of the system
* v.
j
Purpose and scope of the Configuration Management
Plan ~;. j
Related systems affected by Configuration Management
Plan
o Configuration Management Organization
Project Manager
: i f
Configuration ..Manager ' j -
-": Individual a"hd home organization -
i ' '
Change Control Board
-.*'-.' Role and Authority
- Chairperson and home organization
/- Other participants and home organizations
' - Staffing and contractor support (if applicable)
o Configuration Item Identification ^ ?: *
* > ,'* "
Procedure or method for configuration '"item
identification ; , ? .., -;; ^
';-": . .,,. . ' .-. t 'v>
Configuration item indexes for each baseline;1
lists of configuration items and cross-references
among items (The indexes may be an attachment to
Configuration Management Plan)
o .Change Control Process i . ^ ^ *«
Process overview
Change request procedure and form (if applicable)
Change request examination criteria
Levels of examination and approval
Documentation of examination results and approvals
15
-------
EXHIBIT 4-2 (continued)
OUTLINE OF CONFIGURATION MANAGEMENT PLAN
o Configuration Accounting
*"-- Overview of accounting process ?-
T' i ' ' ''.. ' '}
^ -r- Change^request status accounting procedure
Configuration item status forms and procedure
-.; . - ^ ' ! - f .-»( - -,. -,
Configuration status reporting procedure
Automated tools used to support status accounting
*T
-------
OSWER DIRECTIVE #902B.OOa
oo Contents of software release package -
software, documentation
oo Quality control of release packages
oo Transmission or transmittal of package to
each installation
oo New release installation assistance,
Controls to ensure effective system integration
and data base synchronization upon installation
of new software
oo Synchronization of data base at each
-or,-?.;, installation7 " l:
oo Synchronization of data base across
installations for same 'system
9_op Synchronization of data base with related
" ~c: ":data in other systems '
17
------- |