v>EPA
               United States
               Environmental Protection
               Agency
            Office of
            Solid Waste and
            Emergency Response
DIRECTIVE NUMBER:  9028. OOa

TITLE.  OSIR1 S SIS1EH LIFE CICLE MANA01ENT GUIDANCE; PAR! THREE m SIX PRACTICE PAPERS
   •  Project Participation and Coordination; Systei Life Cycle Reviews and Approvals;
     Systei Life Cycle for Expert .Systems; Data Hanapent; Configuration Management;
APPROVAL DATE:

EFFECTIVE DATE:

ORIGINATING OFFICE:

0 FINAL

D DRAFT

 STATUS:



REFERENCE (other documents):
  OSWER       OSWER      OSWER
fE    DIRECTIVE   DIRECTIVE

-------
           unites states environmental protection Mgency
                  Washington. DC 20460
OSWER Directive initiation Request
                                                                     1. Directive Number
                                                                       9028.OOa
                                   2. Originator Information
      vjame of Contact Person
        Asa R.  Frost, Jr.
                   Mail Code
                    OS-110
Office
 OPMT/IMS
Telephone Code
 47506760
      3. Title
       OSWER'S SYSTEM LIFE CYCLE MANAGEMENT GUIDANCE: PART THREE _ SIX PRACTICE PAPERS
       Project Participation and Coordination; System Life Cycle Reviews and Approvals;
       System Life  Cycle  for Expert .Systems;  Data Management; Configuration Management;
      -t.tSoajieiytDf BimBagdmdadB tffiitetaiemeru of purpose)
       The Guidance for  information systems  provides a structured approach for the solution of
       information management problems, particularly those that require consideration of
       automated systems.  The Guidance explains the importance of life cycle management and
      describes the progression of the life  cycle in terms of the activities and products  '
       required for each phase of the life cycle.	.'''"
      5. Keywords
       Automation,  baseline information standards, data, data management, guidance
      6a. Does This Directive Supersede Previous Oirective(S)?    I J(  I
      b. Does It Supplement Previous Directive(s)?
                                                       Yes    What directive (number, title)
                                                       Yes    What directive (number, title) 9028 00
                                                            OSWER1s System  Life Cycle Mgmt.Gc
      7. Draft Level
           A - Signed by AA/DAA
              B - Signed by Office Director
        ! - For Review & Comment
          D - In Development
8. Document to be distributed to States by Headquarters?


Yes
X

No
      This Request Meets OSWER Directives System Format Standards.
      9. Signature of Lead Office Directives Coordinator
                                                    Date
      10.
                                                                      Date
      EPA ForrrMef5-17 (Rev. 5-87) Previous editions are obsolete.
   OSWER           OSWER                 OSWER                O
VE     DIRECTIVE          DIRECTIVE        DIRECTIVE

-------
I"               UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
= •fc""r-                     WASHINGTON, D.C. 20460
                               MAR 2 2  IQOO
                                                            OFFICE OF
                                                    SOLID WASTE AND EMERGENCY RESPONSE
MEMORANDUM

SUBJECT:   Part Three Of OSWER's System  Life Cycle
           Management Guidance - Directive No. 90^

FROM:      Asa R. Frost Jr., Director
           Information Management Staf

TO :        Addressees
                                        fj^n
                                      f F"~
      OSWER  Directive  9028.00   states   that  every  information
 management project  which supports  the Agency's  hazardous waste
 program must  comply with  OSWER's  System Life  Cycle Management
 Guidance.   Parts 1 and 2 of the Guidance were distributed several
 months ago.   Part 3  is reserved for  Practice Papers which describe
 a particular topic  in depth,  reflecting changes in technology or
 organizational structures.  Practice Papers on six topics have been
 completed and  are now being  issued as an addition  to Directive
 9028.00.  The set of Practice  Papers is  available in Room M2416 of
 Headquarters, or  by mail by calling FTS  475-6760.   In addition,
 the set of Practice Papers is being distributed to those who have
 already received Parts 1 and 2 of the Guidance.

      The  Guidance   serves   to  promote  better  management  of
 information  activities  by presenting  a  structured  approach for
 solving problems  and for selecting and using the methods, tools
 and technology appropriate to each  project.   The Practice Papers
 in Part 3 cover the following topics:

      "Data Management During  the Life  Cycle" provides details to
      project managers concerning data modeling, data design, data
      documentation, and such key concepts as data stewardship and
      custodianship. It is the basis for OSWER's data administration
      program.

      "Project Management Plan" describes the  structure and content
      of a key document of the system life cycle.

      "Configuration Management" describes OSWER's practice of
      change control and accounting during the  system  life cycle.

-------
     "Systra  Life Cycle  Reviews  and  Approvals"  describes  the
     process  for  obtaining reviews  and approvals of  life  cycle
     products from the appropriate levels of organization such as
     the OSWER Information Management Steering Committee.

     "System Life  Cycle Paper for Expert Systems" provides guidance
     on the design, development and operational issues for expert
     systems.

     "Project Participation and Coordination" describes the typical
     roles to be filled during the system life cycle and suggests
     which parts of the organizations should fill those roles.

     If you  have  any questions  concerning  the Guidance, or  the
Practice Papers, please call me or Mary Lou Melley at FTS 475-6760.

-------
Addressees (without Attachment)

OSWER Information Management Steering Committee Members
OSWER Division Directors
OSWER Program Management Staff Directors
OSWER Staff Directors,  Office of Program Management and Technology
Headquarters SIRMOS
Inspector General
Assistant Regional Administrators
Regional Planning Coordinators for Hazardous Waste
Regional SIRMOS
ORD Headquarters and Laboratory Managers
OSWER Contractors
OIRM Contractors


Addressees (with Attachment)

OSWER Information Management Coordinators
OIRM Division Directors
National Data Processing Division Director
Regional Waste Management Division Directors
Regional IRM Branch Chiefs
Individual recipients  of  OSWER's  Life Cycle Management Guidance,
         (Parts 1 and  2)

-------
REVIEWS AND APPROVALS

-------
                        OSWER DIRECTIVE #902B.OOa
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
     *             .  •
    Part 3: Practice Paper

      System Life Cycle
   Reviews and Approvals
       January, 1989

-------
                                                      OSWER DIRECTIVE *9323.COe
                        TABLE OF CONTENTS

                                                             Page
1.  Practice Paper Purpose   	    1
2.  Application of Life Cycle Management Guidance	    1
3.  Review and Approval Process	    2
    3.1.  Overview   	    2
    3.2.  Reviews  	    4
    3.3.  Approvals	    6
4.  Threshold Analysis   	-.	    8
    4.1.  Purpose of  Threshold Analysis   	    8
    4.2.  Performing  Threshold Analysis   	    8

                             EXHIBITS

3-1.  Review and Approval  Process for Each Stage	:...   3
4-1.  Criteria for Reviews and Approvals	   9
4-2.  Threshold Analysis Process   	  10

-------
                                                    OSWER DIRECTIVE «5QSB.OOa


                    1. PRACTICE PAPER PURPOSE


    This  practice  paper  constitutes a section of Part 3 of the
Office  of Solid Waste and Emergency Response (OSWER) System Life
Cycle-  Management Guidance.  It describes the review and approval
process  noted  in  Guidance  Parts 1 and 2 in the context of the
OSWER program and information management organizations.

    The topics addressed in this practice paper include:

    o  The  scope of application of OSWER's Life Cycle Management
       Guidance;

    o  The  OSWER  review  and  approval process in a system life
       cycle; and

    o  OSWER1s threshold analysis methodology for determining the
       appropriate review and approval levels for each project.


    OSWER's  process for review and approval of system life cycle
efforts has seven primary purposes:

    o  To confirm the information management requirements  for the
       system:

    o  To  confirm  the quality of the life cycle products from  a
       programmatic,    technical,    and    project   management
       perspective;

    o  To  confirm  that all programmatic, technical, and  project
       management issues have been appropriately resolved, and at
       the proper time;

    o  To  confirm  and document*program management  commitment to
       the   project   and   its   management  approach   (defined
       requirements, schedules, plans, resources, and budget);

    o  To  approve  continuation  with the next stage of the  life
       cycle;

    o  To  confirm  the  continued commitment of resources to the
       project for the remainder of the life cycle;  and

    o  To provide additional feedback to the project team.


        2. APPLICATION OF LIFE CYCLE MANAGEMENT GUIDANCE


    This Guidance applies to systems intended for use by an OSWER
organization,  op by a Regional office or state agency  in support
of  an  OSWER  program.  It includes systems funded  by  the.direct

-------
                                                    OSWER DIRECTIVE #3023.033
OSWER  budget,  or  by  other EPA offices in support of OSWER. It
excludes  systems  developed by state agencies for their own use,
and  systems  developed by OIRM and ORD for Agency-wide use, even
if  they support or otherwise impact OSWER programs or resources.
It   includes  systems  developed  under  OSWER  mission  support
contracts,  even  those  incidental  to  the  major  purpose of a
contract.

    Although  the Guidance has broad application, it provides the
flexibility  needed  to  address  a  wide  range  of  information
systems.    It  does  not prescribe a single method, or present a
"cookbook"   approach   applicable   without   change   to  every
information  management need or system.  Rather, it presents part
of  a  structured,  disciplined  approach for solving information
management  problems,  and  for  selecting and using the methods,
tools,  and  techniques  appropriate  to each problem.  The basic
life cycle management approach is tailored to each system, and is
documented in the Project Management Plan.  (See OSWER Life Cycle
Management  Guidance,  Part 3, Practice Paper; Project Management
Plan.)


                 3. REVIEW AND APPROVAL PROCESS


3.1.  Overview

      Project  activities  and  products  must  be  reviewed  and
approved  at  each  stage  of the system life cycle.  Exhibit 3-1
illustrates  the  structure and timing of the review -and approval
process.    Four  important  aspects  of  the review and approval
process are noted below:

      o  Threshold Analysis determines the appropriate review and
         approval  levels for a system, clearly identifying those
         systems that require approval from the OSWER Information
         Management Steering Committee.  It is first performed in
         the  Initiation phase, and is updated in each subsequent
         phase or stage as needed.

      o  In most stages, a 'project awaiting approval may continue
         to  the subsequent stage.  If the project completes  that
         stage  while the earlier approval is still pending,  work
         stops  until the pending approval is received unless the
         project manager obtains a written waiver.

      o  Before  a project is submitted for approval by the OSWER
         Information  Management  Steering  Committee,  the OSWER
         Senior  Information Resources Management Officer  (SIRMO)
         and   Information   Management  Coordinator(s)   (IMC(s))
         involved in the project conduct the  required reviews and
         document  their  findings.   Based on the  results of the
         review,  the  Project  Manager'  determines   the  specific

-------
                                     EXHIBIT 3-1:
          REVIEW  AND APPROVAL  PROCESS  FOR EACH  STAGE
               BEGIN
               STAGE
.	
}   PARALLEL  I
•    REVIEW   I
|  (OPTIONAL)  I


        \
                        PROJECT
                     APPROACH AND
                       EXECUTION
                       ACTIVITIES
                                  THRESHOLD
                                   ANALYSIS
                                                                       FEEDBACK
                     ""..
                                               PREPARE
                                                SYSTEM
                                               DECISION
                                                PAPER
                                   *••••.
     REVIEW AND APPROVAL THRESHOLDS
             Level I
                           Level
Review

Approval
       IMC(s) and OSWER S1RMO   IMC

       Designated Office Directors)  Office Direclor
        and 1M Steering Committee
a
I  !
a
                                                                                              ri

-------
         changes  or  adjustments to be made, and determines when
         to  proceed to the OSWER Information Management Steering
         Committee to obtain the required approvals.

      o  No  system may be fully implemented (i.e., progress from
         the   Implementation  stage  to  the  Production  stage)
         without formal approval.

3.2.  Reviews
      •"•••^••"••^^^^^^^^•^MV                             4

3.2.1.  Purpose of Reviews

        Reviews are formal examinations of life cycle products to
verify  that  they  solve  the  information  management  problem.
Reviews  are advisory activities, not decision-making activities.
They  are  advisory  to the approval authority and to the project
team.  They have four major purposes:

        o  To  confirm  the  continuing  validity  of  the system
           requirements;

        o  To   confirm   that   the  contents  of  the  system's
           programmatic,   technical,   and   project  management
           products reflect the requirements;

        o  To  confirm  that  all  programmatic,  technical,  and
           project  management  issues  have  been  appropriately
           resolved; and

        o  To provide additional feedback to the project team.

3.2.2.  Identification of Needed Reviews

        Reviews  are  conducted  prior  to  all  approvals.  Each
review  to  be  conducted is identified in the Project Management
Plan,  with  the timing, content, and participants noted for each
review.    Where needed, the content and timing of each review is
tailored to reflect any consolidation of life cycle stages.
                •
3.2.3.  Reviewers
                                                   /
        The  principal  (or  lead)  reviewer is determined by the
results  of  the analysis of the review criteria (see Section 4),
and  is  either  an  IMC  or  the  SIRMO.   When the SIRMO is the
reviewer,  the  IMC(s)  of the program(s) also participate in the
reviews.  Office Directors may establish additional review levels
within their own offices.  The lead reviewer is identified in the
Project Management Plan.

        The  lead reviewer is supported by other organizations or
individuals  (e.g.,  representatives  of OSWER, Regional offices,
state agencies/associations, OIRM, contractors), as determined by
the  reviewer.    The  supporting reviewers are identified in the
Project Management Plan.

-------
                                                      OSWER DIRECTIVE r;2Q22.C2a
3.2.4.  Conduct of Reviews

        Reviews  may be conducted during and at the completion of
project  execution  activities (i.e., the activities performed by
the  project  team),  or both.  Reviews that are performed during
project  execution  activities  may  include  attendance  by  the
reviewer(s)  at key project meetings or presentations,  as well as
an  examination of individual life cycle products. The  timing and
scheduling  of reviews are determined jointly by the reviewer and
Project  Manager,  and  are  documented in the Project  Management
Plan.

        Reviews  examine (directly or indirectly) all products of
the  life  cycle, and always include an examination of  the system
decision   papers.    Reviews  are  carried  out  in  two  steps,
reflecting  the  reviewer's  confidence  in the implementation of
life cycle management and the strength of project management:

        1.  The reviewer examines the* project management products
            to  confirm that the life cycle management activities
            of  the  current phase or stage have been planned and
            conducted   appropriately.    In  such  a  case,  the
            reviewer  may elect to conduct a high-level review of
            most  other  products,  and  examine  only the system
            decision paper in detail.

        2.  In  other  cases,  the reviewer may examine in detail
            the  content  of  one or more of the other life cycle
            products of the current phase of stage.

        Review  results  are discussed with the project team, and
provide  valuable   feedback.   Individual life cycle products are
revised  and  reviewed  as  appropriate  before submission to the
approval   authority.    The  Project  Manager  determines  those
modifications  that are appropriate.  For a project that  requires
the   approval  of  the  OSWER  Information  Management  .Steering
Committee,  the  OSWER  SIRMO  and IMC(s) involved  in the project
conduct  the  review  and  document  their findings.  Although the
Project  Manager* and  the  reviewers should agree  on the changes
needed,  if  any prior to submission of  the system  decision paper
for  approval,  such  agreement  is  not  mandatory.  The Project
Manager  will  determine  the  changes   to  be  made, and when  to
request  approval   from the OSWER Information Management  Steering
Committee.

3.2.5.  Timing of Reviews

        The  length of time required for reviews  varies;  parallel
review  should  be  employed  for  particularly  large projects  or
projects with very  tight schedules.  The duration for each  review
should be determined jointly by the  reviewer and  project  manager.
In  any  case,  review(s) are to be  completed before the  products
are  submitted  to  the approval authority.  For  systems  requiring
OSWER  Information  Management  Steering  Committee approval,  the

-------
review should be completed, and the pertinent products and review
findings  provided  to  the OSWER Information Management Steering
Committee,  at  least  two weeks prior to the scheduled committee
meeting  date  (or the date for written concurrence if no meeting
is scheduled).

3.2.6.  Products of Reviews

        Detailed   review   findings   and   recommendations  are
documented and provided to the project team, either by annotating
individual life cycle products and/or in a separate report.

        Summary  findings and recommendations are documented in a
memorandum  attached  to  the  Decision  Paper  submitted  to the
approval  authority.   If there are significant unresolved issues
regarding  the  findings  or  recommendation  of  the review, the
reviewer clearly notes these issues.

3.3.  Approvals

3.3.1.  Purposes of Approval

        Approvals have four major purposes:

        o  To  confirm and document program management commitment
           to the project and its management approach;

        o  To obtain program management acceptance of the project
           activity results, recommendations, and products of the
           current life cycle stage;

        o  To  approve  continuation  with  the next stage of the
           life cycle; and

        o  To  confirm  the  continued commitment of resources to
           the project for the remainder of the life cycle.

3.3.2.  Identification of Needed Approvals

        Approvals  are  provided in all stages.  Each approval to
be  obtained  is  identified in the Project Management Plan, with
the   timing,  focus,  and  approval  authority  noted  for  each
approval.

3.3.3.  Approval Authority

        The approval authority is determined by the values of the
approval  criteria  (see  Section  4),  and  is  either an Office
Director  or the OSWER Information Management Steering Committee.
Depending  on  the specific requirements of a particular project,
the approval level may be changed or augmented as follows:

-------
                                                      OSWER DIRECTIVE *SC2a.GOa
        o  Office Directors may re-delegate approval within their
           own  offices,   or may elect to elevate it to the OSWER
           Information Management Steering Committee.

        o  The SIRMO has  the authority to elevate approval to the
           OSWER Information Management Steering Committee.

        The  approval  authority  is  identified  in  the Project
Management Plan.

3.3.4.  Conduct of Approvals

        The Project Manager and reviewer will agree on the method
for  conducting  approval  presentations.  The approval authority
may  require  a  formal  presentation  'of some or all of the life
cycle  products  and/or  the  results  of  the  review(s), or may
examine and approve the products without such a presentation.

3.3.5.  Timing of Approvals

        Approval should be timely upon completion of the required
reviews.

        While  awaiting  an approval, a project may continue with
the subsequent stage  (except when progressing from Implementation
to  Production).    If the project completes that stage while the
earlier  approval  is still pending, work stops until the pending
approval is received.

        The  project  may  continue  beyond  the next stage  if so
directed in writing:

        o  by  the  IMC,  for  projects subject to approval  by an
           Office Director, or

        o  by the OSWER Senior IRM Official  (SIRMO), for projects
           subject    to   approval   by   the  OSWER  Information
           Management Steering Committee.

3.3.6.  Products of Approvals

        Approval  of  a specific phase or stage is documented by  a
formal memo, or by signature on the Decision Paper.

        An approval may provide specific direction to the  project
team,  and  modify  the  project  approach,  functional  and data
requirements,  and/or  system  design  presented  in the Decision
Paper.

-------
                                                      CSWER OlfiCTlVs i. £-C£2.C2;-
                      4. THRESHOLD ANALYSIS


4.1.    Purpose of Threshold Analysis

        The  purpose  of  threshold  analysis is to determine the
appropriate project review and approval levels.

4.2.    Performing Threshold Analysis

        Exhibit  4-1  lists  the  criteria  used  in  determining
project review and approval levels/ and their values.

        There are two project review and approval levels:

                   Level I                       Level II

       Review      Information Management        Information
                   Coordinator(s) (IMC(s))       Management
                   and               •            Coordinator
                   Senior Information
                   Resources Management
                   Officer (SIRMO)

                   Level I                       Level II

       Approval    Designated Office    . •        Office Director
                   Director(s) and OSWER  '            .
                   Information Management
                   Steering Committee

        To  determine  the appropriate review and approval levels
for a given project, the Project Manager determines the project's
values  for the criteria listed in Exhibit 4-1.  Any project with
at  least  one  Level  I value is a Level I project.  Exhibit 4-2
depicts the process of threshold analysis.

        Threshold  analysis  is  repeated  in each phase or stage
from  Initiation" through  Implementation,  and documented in the
appropriate  System  Decision  Paper.   ' Repeating  the  analysis
ensures  that a proper level of review and approval is applied to
any  significant  changes  to  the  requirements or design of the
system.    If the appropriate review or approval level changes as
the  result  of  the  analysis,  the  change is documented in the
Project Management Plan and noted in the System Decision Paper.
                                 8

-------
                                                      OSWEH DIRECTIVE i«C2S.C3a
                           EXHIBIT 4-1
               CRITERIA FOR REVIEWS AND APPROVALS
   Criterion
Cost
   Level I Value(s)

Cost exceeds the
available budget
support of the.
sponsoring OSWER
office/ or

Investment cost, annual
operating cost, or
total life cycle cost
meets OMB reporting
requirements*
   Level II Value(s)

Sponsoring OSWER office
can provide sufficient
funding with available
budget support, and all
costs are below OMB
reporting requirements*
Organizational
.Impact/People
Requirements
[for initial
creation of the
system, ongoing
operation after
implementation,
or both]
 Requires FTE  support
 from multiple OSWER
 program offices,  or

 Requires FTE  support
 from Regional offices,
 or

 Requires FTE  support
.from state  agencies
FTE support will be
required of only a
single OSWER program
office; no FTE support
required from other
OSWER offices, Regional
offices, or state
agencies
Data Sharing
 Data  are  created/
 collected/  and/or  used
 by multiple OSWER
 program offices/
 Regional  offices/
 and/or  state agencies
Data are created/
collected, and used
within a single OSWER
program office; no data
are created by or.
collected from Regional
offices or state
agencies
Software/
Technology
Sharing
 Under  consideration as
 a potential OSWER
 standard,  or

 Deviates  from existing
 Agency standards
Not a candidate OSWER
standard, and fully
adheres to applicable
standards
  - OMB  reporting   requirements  are  investment  cost  (Initiation
    phase   through  Implementation stage)  greater  than  or  equal  to
    $5  million, or annual support  costs  greater  than  or  equal  to
    $1 million.

-------
            EXHIBIT 4-2:   THRESHOLD  ANALYSIS  PROCESS
     Cost
    Exceeds
   Sponsoring
Office's Budget or
  Meets OMB
   Reporting
     Req't*
   Requires
  FTE Support
 From Multiple
OSWER Offices,
  egions, and/
    States
     Data
  hared Across
Multiple OSWER
Offices, Regions,
  and/or States
    Potential
    OSWER  \NO
Standard Software
 or Deviates from
    Agency
    tandard
                                   Requires Level I
                                 Review and Approval
                                                      Requires Level II
                                                    Review and Approval
*  Investment costs (Initiation phase through
   Implementation stage) equal or exceed $5 million,
   or annual support costs equal or exceed $1 million.
REVIEW AND APPROVAL THRESHOLDS
Level I
Review IMC(s) and OSWER SIRMO
Approval Designated Office Directors)
and IM Steering Committee
Level II
IMC
Office Director

-------
                      OSWER DIRECTIVE »aQ2Q.QOa
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
    Part 3: Practice Paper
    Project Participation
     and Coordination
       January, 1989

-------
                                                    OSWER DIRECTIVE «90aS.QQa
                        TABLE OF CONTENTS

                                                            Page
1.   Practice Paper Purpose	   1

2.   Roles and Guidelines for Selecting Participants   	   2
    2.1  What is Meant by Project Participant  	   2
    2.2  Program Management  	   6
    '2.3  Program Staff  	   6
    2.4  Project Management  	   8
    2.5  Project Staff  	   8
    2.6  Quality Assurance  	  10
    2.7  Procurement  	  11
    2.8  Timing of Role Designation	  13

3.   Other Factors for Selecting Project  Participants   	  13

                            EXHIBITS
1-1.  Illustration of Roles and Responsibilities of
      Concept Phase	   3
1-2.  Roles and Responsibilities  —  Four Major Groups  	   4
2-1.  Multiple Roles  for System Users   	   7

-------
PROJECT PARTICIPATION

-------
                                                    OSWER DIRECTIVE #9023.003


                    1.  PRACTICE PAPER PURPOSE


      This  practice paper  constitutes a section of  Part 3 of the
Office  of Solid Waste  and  Emergency Response (OSWER)  System Life
Cycle  Management Guidance.  It describes the typical  roles to be
filledin conducting an information system project, and provides
suggestions for determining the organizations that should fulfill
each  role.   It expands on the roles identified in Parts 1 and 2
of  OSWER's  System  Life  Cycle Management Guidance.   This paper
also  suggests  organizations  that  should  participate  in  the
project/ or with whom the project should be coordinated.

      The topics addressed in this practice paper include:

      o  Common   roles   to  which  specific  organizations  and
         individuals  are  assigned to accomplish the  information
         system project;

      o  Relationships  between project roles;

      o  Organizations     that    should    be   considered   for
         participation   or   coordination  in  an  OSWER  system
         project; and

      o  Criteria  to  be  used,  in  determining  the appropriate
        . organizations  from which individuals should be assigned
         to 'each role.                         •

      Proper  attention ' to  project participation serves several
important purposes in support of the system life cycle:

      o  Helps  ensure  that  the  system  focuses clearly on the
         information  management problem and related requirements
         by   identifying   the   appropriate  organizations  and
         enabling   the   most   knowledgeable   individuals   to
         participate actively in the project;

      o  Helps  ensure  that project support continues  throughout
         .the   entire   life   cycle   with  the  involvement  of
         appropriate program managers and likely system users;

      o  Helps  ensure  that  the  system  is  designed well and
         implemented  effectively  by ensuring that organizations
         and  individuals  with  the  proper technical  skills and
         real experience are involved in the project;

      o  Helps  provide clear expectations of the participants in
         a  system  project, particularly of novice participants,
         regarding their respective roles and responsibilities on
         the project.

      This  practice paper discusses project participation  from  a
broad  perspective,  considering  the  participation  of  offices

-------
within   OSWER,   other   Agency   offices,   and   the  external
organizations  (such  as  State  agencies)  that  implement OSWER
programs  and  are  likely  users  of  OSWER  systems.     It also
considers  organizations  within the Agency that may be called on
to  provide  expertise regarding information management analysis,
technologies, and procurement.

      It   should   be   noted  that  the  selection  of  project
participants and the overall organization of the project team are
important parts of the Project Management Plan and are documented
in  the  Project  Management  Plan.  A separate practice paper is
available  that  provides  more  information  about  the  project
management plan.

      This   practice   paper   on   Project   Participation  and
Coordination  does not address*the responsibilities of individual
project participants with respect to the individual activities of
the  system  life cycle.  Rather, it describes the general nature
of  the  responsibilities for the organizations assigned specific
project  roles.    Part  2  of  the  System Life Cycle Management
Guidance  identifies  the  specific activities of the life cycle,
and  the  responsibilities  for  accomplishing them.  Exhibit 1-1
provides   an  example,  taken  from  Part  2  of  the  Guidance,
illustrating   specific   responsibilities  for  several  of  the
activities  of  a  single  phase, the Concept phase.  Exhibit 1-2
illustrates how the roles identified in Part 2 can be viewed more
simply as four groups:

      o  Program Group
      o  Project Team
      o  Quality Assurance Group
      o  Procurement Support Group

      This   practice   paper  does  not  describe  how  specific
individuals should be selected to participate in the project, nor
does  it  focus  on  the  determination  of  potential contractor
support  needs.  Assistance In these matters can be obtained from
the  OSWER  Information  Management  Staff  (within the immediate
office   of   the  Assistant  Administrator),  OSWER  Information
Management   Coordinators   (IMCs),  and  the  Agency  Office  of
Information Resources Management (OIRM).


      2. ROLES AMD GUIDELINES FOR SELECTING PARTICIPANTS


2.1.  What is Meant by  'Project Participant'?

      The  life  cycle  management  process describes a broad and
diverse  set  of activities for solving an information management
problem.    The  activities of the life cycle range from defining
the  problem and selecting the solution, to developing, operating
and  maintaining  the  solution.   Project participants are  those
organizations and individuals that contribute in some substantive

-------
                                   EXHIBIT 1-1
                 ILLUSTRATION OP ROLES  AND RESPONSIBILITIES
                               OF CONCEPT PHASE
                         CONCEPT PHASE
                ROLES AND RESPONSIBILITIES
                              ROLES AND RESPONSIBILITIES
                                                         PROJECT   QUALITY
                                 OSWER PROGRAM
                      KUJC.V.1    vur._...
                      STAFF   ASSURANCE PROCUREMENT
OSWER PROGRAM
                                              MANAGEMENT
     ACTIVITIES
DEFINE AND DOCUMENT
HIGH-LEVEL FUNCTIONAL
REQUIREMENTS DEFINITION

DEFINE AND DOCUMENT
HIGH-LEVEL DATA
REQUIREMENTS DEFINITION

DOCUMENT CONNECTIONS
BETWEEN HIGH-LEVEL
FUNCTIONAL AND DATA
REQUIREMENTS

ASSESS CAPABILITIES OF
CURRENT SYSTEMS, DATA
BASES, AND PROCEDURES
              •
DEFINE ALTERNATIVE
CONCEPTS

 EVALUATE CONCEPTS
 AGAINST REQUIREMENTS
SUPPORT
SUPPORT
SUPPORT
 SUPPORT
 SUPPORT


 SUPPORT
                            LEAD     PERFORM   REVIEW


                            LEAD     PERFORM   REVIEW




                            LEAD     PERFORM   REVIEW




                            LEAD     PERFORM    REVIEW


                            LEAD     PERFORM    REVIEW     SUPPORT

                                          * ,

                            LEAD     PERFORM    REVIEW
                                                                                                 CO
                                                                                                 CD

                                                                                                 8

-------
                                  EXHIBIT 1-2
               ROLES AND RESPONSIBILITIES —  FOUR MAJOR GROUPS
                        CONCEPT PHASE
               ROLES AND RESPONSIBILITIES
     ACTIVITIES

DEFINE AMD DOCUMENT
HIGH-LEVEL FUNCTIONAL
REQUIREMENTS DEFINITION

DEFINE AND DOCUMENT
HIGH-LEVEL DATA
REQUIREMENTS DEFINITION

DOCUMENT CONNECTIONS
BETWEEN HIGH-LEVEL
FUNCTIONAL AND DATA
REQUIREMENTS

ASSESS CAPABILITIES OF
CURRENT SYSTEMS, DATA
BASES, AND PROCEDURES

 DEFINE ALTERNATIVE
 CONCEPTS

 EVALUATE CONCEPTS
 AGAINST REQUIREMENTS
                            ROLES AND RESPONSIBILITIES
DSWER PROGRAM OSWER PROGRAM
MANAGEMENT STAFF
SUPPORT
SUPPORT
1
SUPPORT
. SUPPORT
SUPPORT
.
SUPPORT
PROJECT PROJECT
1ANAGEMENT STAFF
•
LEAD PERFORM
LEAD PERFORM

LEAD PERFORM
LEAD PERFORM
LEAD PERFORM
*
LEAD PERFORM
QUALITY
ASSURANCE

REVIEW
REVIEW

REVIEW
REVIEW
REVIEW

REVIEW
PROCUREMENT






SUPPORT


                                                                                          CD

-------
                                                      OSWER DIRECTIVE #9028.QOa


way  to the evolution of the solution, which often  takes  the form
of an automated  information system and/or data base.

      All  project  participants  need   not  be  experts in ADP  or
information  management technology.  Although having  participants
with  this expertise is vital, other participants may be  selected
due  to  their   knowledge  of the waste  management  program and/or
their project management skills.

      The  selection  of  project participants depends heavily  on
the  nature  of   the information management problem,  the  scope  of
organizations  that  experience  the  problem,   and  a number  of
characteristics   of  the  solution  to   the  problem, such as  the
specific data to be collected and the hardware to be  used (for  an
automated  system).

      This section of the practice paper describes  the roles  of a
typical  system   project  and  provides   general guidelines   for  .
identifying  the candidate organizations that  should  be  called on
to participate   in  a  system   life, cycle  project.   Section 3
identifies   specific  characteristics   of  information management
.problems   and  their  solutions   that   will  help  in  choosing the
appropriate  organizations.

      The  specific  roles described  in  this  section are the  roles
shown in Exhibit 1-1:

      o  Program Management,

      o  Program Staff,
               •
      o  Project Management,

      o  Project Staff,

      o  Quality Assurance,  and

      o  Procurement  Support.

      Project  participants   may  serve  in a  number  of different
 roles   for a given project.    Some  will be members of the project
 team,   either  in the  role  of project  management or project staff.
Others   will  be  involved  in the project to a lesser degree,  but
 still    will  have   an    important   role.     In  selecting   the
participants,    it  is   critical  that   all  participants,  their
 respective  supervisors,   and  the   system project manager have a
 clear and  consistent  understanding  of the participants' extent of
 involvement   in  the  project  (e.g.,  full time,  half time,  one half
day per   week).    This  involvement  should balance  the needs  and
priority   of  the project  with the  other program responsibilities
 the participants will continue to carry.

       It   is important  to  note that the users of the system serve
 in  three  different   roles.     Some users will be members of  the

-------
                                                      uawtH uiHtuuvt
project  team.    Others  will  be members of a Quality Assurance
Group.    The  remaining  users  may contribute to the project by
assisting  the  project  team or the Quality Assurance Group in a
less  formal capacity.  This variety of user roles is illustrated
in Exhibit 2-1.

2.2.  Program Management

      Program  Management  sponsors  and  funds the project.  Key
responsibilities   include   identification  of  the  information
management  problem,  selection  of the project manager and other
members  of  the  project  team,  and approval of the recommended
decisions  and  products  of the project team throughout the life
cycle.

      The  program  management  role  usually is assumed by those
organizations   most   directly   affected   by  the  information
management  problem.   In general, the designated program manager
for  an  information  system  project  will  be  a  member of the
organization  that  will serve as the- primary user of the data to
be  processed  by  the system.  However, some projects have broad
organizational   impact   —  for  these  projects,  the  program
management role must be shared by multiple organizations.

      Candidate  organizations  for this role include regional as
well as headquarters organizations.

2.3.  Program Staff

      Program staff represent the day-to-day users of the system,
and  provide  a  very  strong  programmatic  perspective  of  the
information  management  problem  and  viability of the solution.
Some members of the program staff will be assigned to the project
team.  Other members will have important responsibilities such as
providing  input  and  advice  to  the project team regarding the
functional  and  data  requirements,  reviewing  suggested system
alternatives and the system design, and participating actively in
acceptance  testing •of the system before it is made available to
users.

      Similar  to  the program management role for a project, 'the
designation  of  participants  serving  as program staff reflects
those  organizations  most  directly  affected by the information
management  problem.    The  designation  of  program  staff must
recognize  those  organizations  that collect and/or generate the
data  needed to solve the problem, as well as those organizations
that use the data.

      For  problems with broad organizational impact, individuals
may  be  designated  from  multiple  organizations.    Widespread
participation in this role is common for large systems,  and could
involve    representatives   of   multiple   OSWER   offices   at
headquarters,  other  program  offices, the  regional offices, and

-------
                                        OSWEB DIRECTIVE i?9028.0Qa
               EXHIBIT 2-1:
  MULTIPLE ROLES FOR SYSTEM USERS
PROJECT
TEAM
  Users in an
Assistance Role

       A
                  PROGRAM
                MANAGEMENT
                  PROGRAM
                   STAFF
                     /
                     /
                  EXTERNAL
                   fcNIZATIO
                   local agencies*
QUALITY
ASSURANCE
GROUP
          = SYSTEM USERS

-------
                                                     UOWtK UlHffiHVE 09028.008
state agencies and agency associations.  In some instances it may
be appropriate to involve industry as well.

2.4.  Project Management

      The   project   management   role  provides  direction  and
management   of   the   system   project,  and  retains  ultimate
responsibility  and  accountability  for  solving the information
management problem within the approved timeframe and budget.

      Each  system  project  has  a  single  Project  Manager, an
individual  drawn  most often from the organization(s) sponsoring
the project.  The project manager may be drawn from a regional or
headquarters organization.

      For • a  very  large  system project, such as for a national
system  involving  several  OSWER  offices  and  the regions, the
project management role may be performed by group of individuals.
The establishment of a project management group helps ensure that
a broad programmatic and organizational perspective is applied to
key   analyses  and  to  recommendations  to  program  management
throughout  the life cycle.  For these projects, a single Project
Manager  is  designated  to  manage  the  day-to-day  work of the
project,  and  is  a  member  of  the  project  management  group
(referred to by a name of its choosing).  The relationship of the
Project  Manager  to  this group is established on a case by case
basis by the program manager.  For some projects, this group will
be  advisory to the Project Manager.  For others, this group will
have decision authority and will direct the Project Manager.

      The  selection of the organizations) and individual(s) for
the  project  management  role  is critical to the success of the
project.   This selection should carefully consider, and balance,
the following factors:

      o  Knowledge  of program policy and operations commensurate
         with the scope of the information management problem;

      o  General project management ability, including management
         .of internal personnel and contractors (if appropriate);

      o  Information system project management ability;

      o  Expertise  in  information management methods, tools and
         technologies; and

      o  Expertise  in  conducting  procurements  of  information
         management technology or services  (if applicable).

2.5.  Project Staff

      Project  staff  perform the majority  of project activities,
working  under  the  direction  of  the  Project  Manager.   These
participants provide the full range of programmatic and technical

                                8

-------
                                                     OSWEfl DIRECTIVE #902B.OOa



knowledge,   skills,   and  abilities  needed  to  accomplish  the
project,   using   individuals   drawn   from   programmatic  and
information management organizations as appropriate.   Quite often
contractors   serve  as  members  of  the  project  staff.    The
organizations  and  individuals  filling  this  role,   along with
Project  Management,  are commonly referred to collectively as the
'project team1.

      The  Project  staff bring to the project team a  combination
of  the  knowledge,  skills and experience needed to successfully
analyze the problem,  select an appropriate solution,  and develop,
implement and maintain the solution until it is no longer needed.
The  Project  Staff  brings  together  many  areas  of expertise:
programmatic,    systems    analysis,    information   management
technology, and other areas.

      The  organizations and individuals selected for the Project
Staff   should  provide  the  needed  abilities,  and  represent,
collectively,   the   organizations   who  are  experiencing  the
information management problem or will be involved in developing,
operating, and/or maintaining its solution.  Project teams should
always  include  representatives of user organizations, including
organizations  that  will  use  the  information  provided by the
system,  and also organizations that will generate and/or collect
the  data to be processed by the system.  Thus, the project  staff
often  includes  individuals outside of OSWER for a headquarters-
based  project,  and  individuals,  outside  of the regional  Waste
Management Division for a single-region project.

      The  selection  of  the  Project  Staff  is usually a  joint
effort  .of  the  Project Management and Program Management roles.
Where  coordination  across  program  or  organizational  lines is
needed  (e.g.,  selecting  team  members  from Regional offices),
Program  Management  usually  has  lead  responsibility   for such
coordination  to  ensure  that  individuals  are  assigned at the
proper level and with the appropriate extent of commitment  (i.e.,
level of effort) to the project.

      For  problems with broad organizational impact,  individuals
may  be  .drawn from many organizations.  Widespread participation
on  the  Project Staff is common for large systems.  Listed  below
are  those  organizations  who may be called on to participate as
members of the project staff.

      o  Individual OSWER program offices,

      o  Regional Waste Management Divisions,

      o  Regional Environmental Services Divisions,

      o  Regional Enforcement Divisions,

      o  Regional Management Divisions,

-------
                                                    OSWER DIRECTIVE #9023.003
      o  Other Agency program offices,

      o  Office of Information Resources Management,

      o  OARM/RTP - National Data Processing Division,
                                               t
      o  Other Federal agencies,

      o  State waste program management agencies,

      o  Associations of State agencies, and

      o  Industry and industry associations.

      Section  3  of  this  practice  paper  provides  additional
guidance to help in selecting the project staff.

2.6.  Quality Assurance

      The  quality  assurance  role  conducts informal and formal
reviews  of system life cycle products, providing feedback to the
project  team  and advice to Program Management regarding project
approvals.  The quality assurance role for a project is performed
by   individuals   representing   programmatic   and  information
management  organizations,  with  many  of  the  same  skills and
abilities   of   Project  Management  and  Project  Staff  roles.
However,  the  individuals selected for quality assurance are not
members of the project team as defined above.  The organizational
level  for  performing quality assurance, particularly for formal
project  reviews,  depends  on the threshold level of the system.
(The   OSWER   formal  review  and  approval  process,  including
threshold analysis, is explained in the practice paper on 'System
Life Cycle Reviews and Approvals'.)
                                                    i
      Individuals assigned to the Quality Assurance role would be
drawn  from  many  of  the  same  organizations  as  the  Project
Management  and  Project Staff.  However, individuals assigned to
the quality assurance role cannot also be assigned to these other
roles.    Doing  so  would result in a conflict of interest, with
these  individuals  ostensibly performing an "independent" review
of  their  own  work,  or  of  the  work  of  their  project team
associates.    It  should also be noted that although individuals
assigned  to  the Quality Assurance role, Project Management role
and Project Staff role are drawn from the same organizations, far
fewer individuals are assigned to the Quality Assurance role than
to the combination of Project Management and Project Staff.

      Leadership  of  the  quality assurance role (referred to as
'lead  reviewer')  varies, depending upo. the level of review and
approval, required  -for the project.  This  level is determined by
.an  analysis  referred  to  as   'Threshold  Analysis'.  The OSWER
senior IRM official serves as lead reviewer for a Level I system,
and  an  Information  Management Coordinator (at headquarters) or
another  seasoned  information  management  professional serves as

                               10

-------
                                                     OSWER DIRECTIVE 09028.00a
lead  reviewer  for  a  Level  II  system.  The practice paper on
'System   Life   Cycle   Reviews  and  Approvals'  describes  the
distinction  between  these two levels of system projects in more
detail.

      It  should  be noted that some projects, particularly those
that  are  very  large (over $1 million cost for Initiation phase
through  Development and Implementation Phase) or will have major
impact  on  program  operations,  may at some point in their life
cycle be subject to a review or audit by the Agency Office of the
Inspector  General  (OIG).  The interests and concerns of OIG may
have  a  major  impact  on  the  overall  schedule  and  cost  of
completing  the  system  if  not  identified  and resolved at the
proper  point of the life cycle.  Obtaining OIG participation (in
a  Quality Assurance role) early in the life cycle would minimize
subsequent   disruptions  due  to  suggested  changes  and  their
resolution.    With  early  resolution, changes will have minimum
impact  due  to OSWER's ability to incorporate the changes in the
products  of the current phase/stage, and by updating the Project
Management  Plan   for future phases and stages.  Program Managers
are therefore strongly encouraged to  request OIG participation in
these projects at  their inception.

      Should OIG decline to participate,  the OSWER/AA Information
Management  Staff  should be informed.  Several potential actions
will be considered:            .

      o  Sending   copies  of  approved  system decision papers to
         OIG, and

      o  Contacting OIG at the beginning  of each of the following
         phases  and stages to participate in a Quality Assurance
         role:

         —  Concept phase
         —  Definition Stage
         —  Design stage
         —  Implementation stage
         —  Evaluation stage

The  actiori(s) taken should be  recorded  in the Project  Management
Plan.

2.7.  Procurement

         The  Procurement  role  provides expert  advice   to the
Project  Manager   in  planning   for   the  acquisition   of   needed
resources other than EPA personnel.   These resources  include ADP
and  communications hardware and software, and contractor  support
services.    The   procurement   role also  participates  actively in
(often leading) other procurement activities  of  individual  system
projects.
                                11

-------
                                                     OSWER DIRECTIVE #902B.OOa

      Although in many cases the needed resources can be obtained
via  existing  contracts,  in  some cases it will be necessary to
conduct  a  new  procurement.   Both of these methods of resource
acquisition  require  specialized  knowledge,  and conducting new
procurements  (when  necessary) requires exceptional expertise to
successfully navigate the intricacies of the procurement process.

      As  soon  as  a  project  identifies  the need for resource
acquisition, it is essential that the Project Manager contact the
pertinent   procurement   support   organization  for  additional
information  and insight.  For most projects/ the need to acquire
resources  will be identified as part of the system concept.  For
large  projects  in  particular,  the  need to acquire contractor
support may be evident early in the Concept phase, or as early as
the Initiation phase.

      Several   organizations   may  provide  procurement-related
support   to  the  Project  Manager.    However,  only  the  OARM
Procurements and Contracts Management Division (PCMD) can legally
commit  the  Agency to a new contract or modify the provisions or
terms  of  an  existing contract.  The other organizations listed
below  can  serve in an advisory capacity to the Project Manager.
In  addition,  some  of  them are required to approve procurement
requests before they are processed by PCMD.

      Some  projects  will  acquire  different types of resources
and/or  require  the  use  of multiple contracts or procurements.
For  these  projects,  individuals  may  be  drawn  from multiple
organizations,  to  handle  multiple  procurements.  The specific
organizations  to  be contacted within the Agency for procurement
advice and support for a given project may include:

      o  OARM/Procurement and Contracts Management Division
                                                    t
      o  Regional  Management Divisions, IRM Branch (for regional
         projects), *

      o  OSWER program office Information Management Coordinators
         (IMCs) (for headquarters projects), *

      o  OSWER/AA Information Management 'Staff, *

      o  Regional   Waste   Management   Division   IRM  Planning
         Coordinators (for regional projects), *

      o  Office of Information Resources Management  (OIRM),

      o  OARM/RTP - National Data Processing Division, and

      o  Other  Federal agencies with interagency agreements with
         EPA,  or  applicable  'schedule1 contracts  (e.g., General
         Services Administration).
                                12

-------
                                                      OSWER DIRECTIVE #902B.OOa


      Project  Managers  should  contact one of the organizations
designated  above  with  an  asterisk (*) before contacting other
organizations for procurement support.

2.8.  Timing of Role Designation

      To   ensure   a  successful  solution  to  the  information
management  problem, it is important that all participants in the
project  understand  what  their  role  is, how they fit into the
overall  effort  to  solve  the  problem.   Within the life cycle
process,  the  roles of Program Management and Project Management
are  assigned  during  the Initiation phase.  All other roles are
assigned  as  early  as  possible  in  the next phase of the life
cycle,  the  Concept  phase.    These assignments are refined, as
appropriate,  throughout the remainder of the life cycle based on
the experience of the project.


       3. OTHER FACTORS FOR SELECTING^ PROJECT PARTICIPANTS

      The  selection  of  specific  organizations  for  the roles
described  in  Section  2  above  should  be  sure  to •obtain the
participation  of  organizations  with essential skills, insights
and  experience.  The listing presented on the following pages is.
intended   to  aid  Project  Managers  and  Program  Managers  in
identifying  the  appropriate  organizations  to participate in  a
particular project.
                               13

-------
                                                      USWEH DIRECTIVE »JU28.QOs
         FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
    FACTORS PERTAINING TO THE INFORMATION MANAGEMENT PROBLEM
Information required from State,
County or Local level
organizations.
o  OIRM - State/EPA Data
   Management Project

o  Regional Waste
   Management Divisions

o  Professional
   associations such as
   NGA, ASWPCA, ASTSWMO
Information required from EPA
Regional offices.
o  Regional Waste
   Management Divisions

o  Regional Environmental
   Services Divisions

o  Regional Management
   Divisions

o  Regional Enforcement
   organizations
Some of the required information is
classified/ sensitive or can be
claimed .as confidential business
information (CBI).
o  OSWER/AA Information
   Management Staff

o  Office of Information
   Resources Management

o  Facilities Management
   and Services Division
   (Security Unit)
Some of the required information  is
generated or collected by another
EPA program office (external  to
OSWER).
   Senior Information
   Resources Management
   Official (SIRMO)  for
   that program office

   Office of Information
   Resources Management/
   Program Systems
   Division
                                14

-------
                                                     QSWEfl DIRECTIVE #9028.0Qa
         FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
        FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
Information will be collected
directly from State, County or
Local level organizations.
   OIRM - State/EPA Data
   Management Project

   Regional Waste Management
   Divisions
Information will be submitted by
EPA Regional offices.
o  Regional Waste Management
   Divisions
Any facet of system operation will
require FTE or contract resources
at the regional offices.
o  Regional Waste Management
   Divisions

o  Regional Management
   Division
Any facet of maintaining the
application software for the system
will be performed in the regions
(e.g., regional or state
enhancements, installation of
software upgrades/new releases from
headquarters)
o  Regional IRM Branch
Part of the system will operate on
the EPA mainframe computers at the
NCC and/or WIC.
o  OARM/RTP - National Data
   Processing Division

   —  National Computer
       Center (NCC)

   —  Washington
       Information Center
       (WIC)
Part of the system will use  the EPA
telecommunications network
o  OARM/RTP - National  Data
   Processing Division
Part of the system will operate on
the regional logical mainframe
computers
o  OARM/RTP  - National  Data
   Processing Division

o  Regional  IRM Branch
                                15

-------
         FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
        FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
                           (Continued)
Part of the system will operate on
the regional minicomputers
o  Regional IRM Branch
Part of the system requires the
installation of new computer
equipment  (including personal
computers), or needs office
environmental conditioning
equipment.
o  OIRM at headquarters;
   Regional IRM Branch

o  Facilities Division (at
   headquarters or regional
   office, as appropriate).
Part of the system requires the
installation of a local area
network (LAN), or the modification
of an existing (LAN) or any other
wiring (e.g., power supply,
telephone lines).
Part of the system will use shared
OSWER minicomputer capacity (e.g.,
Prime minicomputers).
o  OIRM at headquarters;
   Regional IRM Branch

o  OARM/RTP - National Data
   Processing Division

o  Facilities Division (at
   headquarters or regional
   office, as appropriate).

o  OSWER/AA Information
   Management Staff
Part of the System will submit or
receive data from the Agencies
Financial Management System or
Document Control Register System
(including Payroll System).
o  OIRM Administrative
   Systems Division

o  Agency Financial
   Management Division
Part of the System will process
personnel data (potentially subject
to Privacy Act requirements)
o  Office of Administration,
   Personnel Management
   Division

o  OIRM Information
   Management and Services
   Division
Part of the System will use
software that is not an Agency
standard, as defined by NDPD.
o  ^SWER Program Office
   Information Management
   Coordinator (for
   headquarters)

o  Regional IRM Branch
                               16

-------
         FACTORS FOR SELECTION OF PROJECT PARTICIPANTS:
        FACTORS PERTAINING TO THE DESIGN OF THE SOLUTION
                           (Continued)
Part of the system, including
manual procedures, relates to
procedures for handling documents
or other artifacts that are
considered 'official record
material' (includes use of digital
imaging technology as well as
microfilm, microfiche, or other
hardcopy media).
OSWER/AA Information
Management Staff

OIRM Information
Management and Services
Division
Part of the system will use expert
system technology.
OSWER/AA Information
Management Staff
High volume printing of
documentation is necessary
Printing unit in
Facilities Division, OARM
(or regional counterpart)
                      OTHER CONSIDERATIONS
New information will be collected
by the Agency from sources outside
the Agency  (OMB approval  is
required)
Office of Policy,
Planning and Evaluation,
Office of Standards and
Regulations
                                17

-------
                        OSWER DIRECTIVE #9028.008
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
    Part 3: Practice Paper
      Expert Systems
       January, 1989

-------
                                                               OBWER DIRECTIVE *902B.OOa
                              TABLE OF CONTENTS
Chapter 1  Expert Systems Defined

1.0  Introduction	1-1
1.1  Expert System Capabilities.	1-3
1.2  Expert System Benefits	1-4
1.3  Expert System Limitations	1-4
1.4  Differences Between Decision Support Systems,
     and Management Information Systems	'.	1-5
1.5  Expert System Components and Architecture	1-8
1.6  Application Types	1-9
1.7  Prototyping Overview	1-12
1.8  Sources of Additional Information	1-14
1.9  Quick Briefing on Expert Systems	1-15
Chapter 2  Implications of Using Expert Systems at EPA

2.1  Expert Systems for Policy Interpretation	2-1
2.2  Legal Issues	..2-1
2.3  Organization's Culture	2-4
2.4  Cross Cutting Considerations	2-6
2.5  Resource Requirements	2-11

Chapter 3  Initiation Phase

3.0  Introduction	:. .3-1
3.1  Objectives	3-2
3.2  Decisions	3-2
3.3  Success Factors	3-3
3.4  Products	3-5
3.5  Activities	3-7

Chapter 4  Concept Phase

4.0  Introduction	4-1
4:1  Objectives	4-2
4.2  Decisions	4-2
4.3  Products	4-5
4.4  Success Factors	4-7
4.5  Activities	4-10
4.6  Knowledge Management	4-13
4.7  Feasibility Assessment	4-17
4.8  Knowledge Representation	4-17
4.9  Control Structures	4-19
4.10 The System Concept	..-.	4-20
4.11 Proof-of-Concept Prototype	4-20
4.12 Assigning a Project Team	4-20

-------
                                                                 OBWER DIRECTIVE #9028.UUa
Chapter 5  Definition and Design Phase

5.0  Introduction	5-1
5.1  Obj ectives	5-1
5.2  Decisions	5-1
5.3  Products	1	5-2
5.4  Success Factors.	5-3
5.5  Selecting a Development Environment	5-5
5.6  Developer Qualifications	5-12
5.7  Design	5-13
5.8  Prototyping Steps	5-15

Chapter 6  Development Stage

6.0  Introduction	6-1
6.1  Objectives	6-1
6.2  Decisions	6-1
6.3  Products	6-1
6.4  Success Factors	.'.6-3
6.5  Activities	6-5
6.6  Knowledge Acquisition	6r6
6.7  Collection Methods	6-7
6.8  Resolving Conflicts.	6-10
6.9  Knowledge Notebook	6-11
6.10 Prototyping Steps	6-11
6.11 Testing and Validation	6-12

Chapter 7  Implementation Stage

7.0  Introduction	7-1
7.1  Obj ectives	7-1
7.2  Decisions	7-1
7.3  Products	•		7-2
7.4  Success Factors	;	7-3
7.5  Activities	7-5
7.6  Distribution Strategy	7-6

Chapter 8  Operation Phase

8.0  Introduction	8-1
8.1  Objectives	8-1
8.2  Decisions	8-2
8.3  Success Factors	8-3
8.4  Products	8-4
8.5  Activities	8-6

Appendix A  Evaluating Expert System Shells	A-l

Appendix B  Glossary	B-l

-------
                                                     OSUER Directive #9028.OOa
                                  CHAPTER 1
                           EXPERT SYSTEMS DEFINED
1.0       INTRODUCTION

          The purpose of this  practice  paper  is  to  help software  developers
and project managers in EPA's Office of Solid Waste & Emergency Response
(OSWER) understand the issues involved in successfully applying expert systems
technology to OSVER information management problems.

          This paper explains  when and  where  expert systems can be used and
how the development of expert systems should be managed in OSWER.  It is an
extension of the OSWER System Life Cycle Management.  The paper focuses on the
expert system life cycle as it relates to the conventional life cycle phases.
For this Practice Paper to be used effectively, the project leader and
developer should have a firm understanding of the OSWER System Life Cycle
Management Guidance, or LCM Guidance as it will be called elsewhere in this
document.  Further, this paper does not explicitly define or describe in
detail how to build an expert system.  The reader  is responsible for obtaining
this information through formal training, seminars, computer literature (see
the bibliography in Section 1.8), or other media.  The end of CHATTER 1 has a
Quick Briefing on Expert Systems.

          The paper is divided into the eight chapters.  Those specifically
related to the LCM Guidance (CHAPTERS  3 through 8) should be read  in sequence.
When reading the chapters,  bold-face  terms are products associated with each
of the LCM phases.  The first occurrence of terms  defined in the glossary are
bracketed like this   [term].  Each chapter has several additional  features in
common with the rest:

          o        An introduction;

          o        A statement of objectives;

          o        A list  of  the  decisions that must be made  in  the life
                    cycle phase;

          o        A list  of  success factors; and

          o        A description  of activities  that are to be  conducted
                    during  each phase or stage.

          Each chapter contains  an illustration of the objectives, decisions,
and products for the phase.  The  illustration at the start of each chapter
provide the reader with an orientation point in each chapter.
                                      1-1

-------
                                                     OSWER Directive #9028.OOa

          The chapter you are reading, Expert Systems Defined,  introduces the
concept of [expert systems].  It explains the capabilities, benefits, and
limitations that may be associated with their use.  It provides examples of
typical expert system applications.

          CHAPTER 2;  implications of Using  Expert Systems,  gives an overview
of the possible impacts expert systems could have on OSWER.  It identifies
possible legal and liability issues as well as potential impacts on Agency
policies.  The chapter describes the cross-cutting project management issues
that are addressed in multiple phases of the system life cycle.  These are
project management plan, project participation, reviews and quality assurance,
project approvals, configuration management, data administration,
methodologies and [tools], cost-benefit analyses, and [knowledge management].
Finally, the chapter addresses the resources required throughout the expert
system life cycle.

          CHAPTER 3;  Initiation Phase, describes the tasks  involved in problem
definition and in determining the need for an automated solution.  This
chapter explores characteristics of a problem that suggest a possible expert
system solution.

          CHAPTER 4;  Concept Phase,  depicts the identification  of a feasible,
timely, cost-effective solution to the problem.  This chapter contains
information on  the use of proof-of-concept prototyping to refine the
solution, [knowledge representation], and management techniques, expert system
[control structures], and system development justification approaches.  Also,
it specifies required qualifications for system developers.

        .  CHAPTER 5;  Definition and Design  Phase,  discusses issues as they
relate to confirming the suitability of the  [System Concept] and determining
detailed functional requirements.  -It discusses all aspects of selecting a
development environment including [knowledge base] creation, migration to
delivery environments, and user interfaces.

          CHAPTER 6;  Development Stage,  describes issues in developing the
expert system.  This chapter introduces various  [knowledge acquisition]
methodologies, sources, and  [conflict resolution] techniques.  Also, it
defines the means and importance of testing and validating the system.

          CHAPTER 7,  Implementation Stage,  identifies the strategies for
distributing expert systems.  It includes the issues of beta testing, user
registration, hardware and software requirements, training, licensing,
documentation, configuration management, and version control.

          CHAPTER 8,  Operation Phase,  focuses on the Production, Evaluation,
and Archive Stages of the life cycle.  It describes expert system maintenance,
end-user support requirements, knowledge revalidation options and maintenance,
ongoing training and documentation, and software updates.
                                      1-2

-------
                                                     OSUER Directive #9028.OOa

          The paper concludes with two appendices.   Appendix A describes  the
process of evaluating [rule-based] expert system software packages (e.g.,
[shells]) and provides an evaluation form.  This appendix will be useful to
project managers and software developers during the [Definition and Design
Phase] when the development environment may be a rule-based shell.  The
process of evaluating shells using other knowledge representation schemes
(e.g., [frames], object-oriented programming, etc.) has not been defined.
This type of information is widely available in the professional literature.
See Section 1.7 for references on further reading.  Appendix B is a glossary
that provides a definition of the [artificial intelligence] terms and concepts
used throughout this practice paper.


1.1       EXPERT SYSTEM CAPABILITIES

          In order to understand how expert systems can be used
advantageously, the project manager or software developer must first
understand how knowledge processing in expert systems differs from
conventional data  processing.  Expert systems are unique in their ability to
process  [knowledge], not just data.  Knowledge processing differs from data
processing in the  type of information used,  the techniques  used to analyze the
information, and in the form that the results of the knowledge processing are
presented to the  [user].

          Conventional systems limit the developer to data representation
using  only numbers and.text.  They process  data using complex  [algorithms]
that complete a discrete number of steps  to reach a predetermined conclusion.
Expert systems permit knowledge representation -- the encoding of human
decision-making processes using symbolic  terms or  [symbols].  Because expert
systems  process knowledge, they are often referred  to as  [knowledge-based
systems].

          The ability to represent knowledge in symbolic terms expands the
range  of analysis  techniques  that computers can apply to information thus
enabling a system  to emulate some aspects of human  performance.   The expert
system uses  problem solving  procedures such as pattern-matching to reason
about  the symbolic terms.  An example  of  [symbolic  reasoning] by an  expert
system that  determines the daily  forecast is,  "if the sky is cloudy, then the
forecast might  include rain."  This phrase  could be used within an expert
system.   A conventional system would  require that the symbolic terms such as
"sky"  and "cloudy" be defined concretely  as numbers or  text.  The expert
system uses  pattern-matching on the phrases "sky is cloudy" and "forecast
might  include rain" to reach a conclusion about the forecast without requiring
definition of each of the terms within the  phrases.  A  conventional  system
would  require that the developer  define a set number of steps to,determine the
daily  forecast.  The expert  system examines all possible .solutions using the
problem  solving procedure that has been programmed  into it.
                                       1-3

-------
                                                     OSWER Directive #9028.OOa

          The combination of problem solving procedures chat are built into
expert systems, together with the developer's ability to define problems using
symbolic terms, gives expert systems the capability to store and manipulate
more complex relationships between individual pieces and groups of information
than can be accomplished with the processing supported by conventional
systems.

          In addition to knowledge representation,  expert systems also provide
the capability to simplify the user/computer interaction.  Expert systems can
be programmed to employ more of the conventions that people use when
communicating with each other.  Expert systems can be designed with the
ability to explain the "reasoning" used in reaching a recommendation and to
justify their approach to a problem, much as people do.  The more
sophisticated expert systems employ a [natural language] processor to permit
users to consult with the system using near-English rather than structured
query languages or menus.


1.2       EXPERT SYSTEM BENEFITS

          Expert system applications take  advantage of the  above capabilities
in two ways.  First, information becomes more accessible so people can make
better decisions.  Second, where useful information is accessible, people can
be more productive.  OSWER can be more productive in the use of its personnel,
funding, and information resources through the application of expert systems.
The two major benefits associated with expert systems are better decision
making and increased productivity.  These benefits are described below.

          1.2.1	Better Decision Making

          Expert systems improve the quality of decision making by providing a
mechanism for pooling the.knowledge of multiple experts and making that
knowledge available to a. wider audience.  This leveraging of knowledge results
in improved quality of complex work products such as permits, technical
reports, and analyses that recommend actions.

          Expert systems establish a basis for defensible decision-making by
capturing and applying knowledge in verifiable form.  For example, in
developing work products such as management reports and environmental
analyses, a given set of inputs, no matter how complex, should result in
consistent results given closely similar data, advice, or recommendations.   In
addition, the process of reaching a conclusion can be explicitly demonstrated.
This will ensure consistency in many decision-making activities.

          1.2.2	Increased Productivity

          Expert systems offer significant, measurable increases in
productivity by effectively capturing the knowledge of experts and by
minimizing the loss of  [expertise] and knowledge due to attrition.  Expert


                                      1-4

-------
                                                     OSUER Directive #9028.OOa
systems provide a means of extracting, storing, and sharing knowledge in a
variety of disciplines.  Thus, more people have access to expertise.  In turn,
the experts are freed from relatively mundane tasks to focus on demanding
ones.
1.3       EXPERT SYSTEM LIMITATIONS

          Expert systems provide valuable new capabilities,  but they also have
clear limitations.  As with all new technology, developers must weigh the
limitations associated with the use of expert systems technology against
benefits.  Because expert systems emulate human performance in decision
making, they may be incorrectly thought of as having the capacity to make
independent judgments.   Expert systems are capable of communicating advice
that has been coded into them.  They are not capable of producing independent
decisions.  Their application is limited to strictly defined domains (i.e.,
areas of expertise where boundaries on what .expertise should be included  in
the system can be defined); their performance degrades dramatically when
dealing with information that is beyond those boundaries.

          Expert systems can manipulate only symbolic information, that is,
all "real" information  that is collected by observing an event.  For instance,
temperature and humidity in the case of weather forecasting must be translated
into a form acceptable  to  the expert system.  Any errors and biases
incorporated in the translation process will be accepted by the expert system
without question.   An  example of  a translation bias is if temperature
measurements input to the  system are in Fahrenheit when the logic or knowledge
encoded into the system is based on Celsius measurements, then the conclusions
reached will be invalid.

          Another limitation is the difficulty developers might experience
when linking expert systems to conventional data processing systems.  This
would involve building  a hybrid or embedded system.  New expert system
products are appearing  regularly to bridge the gap between expert and
conventional systems.   However, until a greater number of these products
exist, software developers will experience difficulty making expert system
software communicate with  conventional software.
 1.4      DIFFERENCES BETWEEN DECISION SUPPORT SYSTEMS,
          MANAGEMENT INFORMATION SYSTEMS, AND EXPERT SYSTEMS

          Based on an understanding of the capabilities, benefits, and
 limitations of an expert system,  the project manager or software  developer
 needs  to consider the  differences between expert systems and conventional
 systems  such as  database management systems.  The following discussion will
 provide  background needed to determine whether  expert  systems are the
 technology of choice for an application  under consideration.
                                       1-5

-------
                                                     OSUER Directive *9028.00a

          All software tools and applications bear fundamental structural
similarities to each other; their differences lie in the capabilities
associated with each and in the methods or mechanisms used to achieve those
capabilities.  Expert systems can be distinguished from conventional systems
based on those differences.  For example, conventional systems do not have the
capability to handle uncertainty.  They are developed using high level
programming languages such as COBOL or Fortran, and these languages must
explain in exacting detail how to solve a given problem.  They provide no
capacity for vagueness or uncertainty.

          Expert systems accommodate  uncertainty through weighting schemes.
These weighting schemes calculate certainty levels by determining if a piece
of information is correct or valid and increment the level of certainty
accordingly.  The inability to process uncertain data* confines the power of
conventional systems to linear programming tasks, that is,  tasks where steps
are performed in sequence regardless of the inputs, and which are number- or
text-intensive.  While conventional systems are developed using conventional
languages, expert systems are often developed with software referred to as
"shells."  A shell provides a framework to build expert systems.  Appendix A
provides a framework for evaluating shells.

          Expert systems are also developed using LISP  or [PROLOG].   These  are
the recognized computer languages of artificial intelligence (AI).  These
languages simplify the expert system development process because they are
interactive, and they provide feedback to the programmer as the program is
being written.  This capability allows the programmer to see results more
quickly.  Additionally, LISP and PROLOG code is generally more like English
and can be understood more readily than can the standard code of high level
languages.  AI languages give the programmer the added capability of more
flexibility.  Whereas LISP and PROLOG permit programmers to tell the computer
what to do, conventional languages require them to tell the computer how to do
it.  With LISP and PROLOG', less development time is spent defining each piece
of information that might enter the system.  The distinction between use of
programming languages and expert system shells is that the [inference engine]
in the shell has already been validated by the vendor.   Shells are good places
to start prototyping work, and the developer can move to a programming
language if the complexity of the task warrants building the inference engine
from the ground up.

          There are additional differences between expert and conventional
systems.  These are described below.

          1.4.1	Decision Support Systems

          Decision support systems (DSS)  allow users  to combine their judgment
with data to manipulate the data and produce reports.  The users' judgement 5s
translated into an algorithm that is then programmed into the decision support
system using conventional programming languages such as COBOL or Fortran.  'The
algorithm typically gives the user flexibility to define data manipulation


                                      1-6

-------
                                                     OSUER Directive #9028.00«

scenarios and report formats.  A specialized interface allows the user to
easily define meaningful data manipulation scenarios without requiring an
understanding of the underlying data structures.

          DSSs employ a variety of information management  technologies to
solve structured or unstructured problems and support managerial judgment,
thus improving the effectiveness of decision-making.  Typical DSS tools
include spreadsheet, geographic information systems, modeling packages, and
statistical packages.  These systems are also linked to the functions of 4th
Generation" relational database management software.

          Expert systems are a specialized subset of DSSs  that employ symbolic
reasoning to manipulate data and produce reports.  If the judgement employed
by the users is procedural  in nature (i.e., requires a set number of  steps to
reach a predetermined conclusion), then a conventional DSS should be
developed.  However, if the user reaches a conclusion based on a variety of
factors that cannot be captured using an algorithm, then an expert system
could be developed.

          1.4.2	Management  Information  Systems

          Management information systems are computer-based information
systems that provide information  to support management activities and
functions.  They support four  general types of  activities: transaction
processing such as payroll,  sales  order, and  inventory; operational control
which ensures  that activities  are  carried out efficiently and effectively;
management control which measures  performance,  deciding on control actions,
formulating new decision  [rules],  and allocating resources); and, strategic
planning which involves developing strategy with which to meet organizational
objectives.

          Executive information systems (EIS) are a specialized form of
management information  system.  They offer top  level managers access  to
corporate information summarized  in a manner  specified by the executives so
they get what  they need in  the required form  and detail.

          Management information systems do not typically have symbolic
reasoning capabilities.  The graphic displays are  static in the sense that
there is no  [explanation] or interpretation facility.  The user must  know how
to format queries;  the  system  cannot reformat queries if the user is  not
sufficiently  specific.  Vague  or  uncertain information contained in a
management information  system  query may result  in  computer resource intensive
processing caused by extensive database  [searches]).
                                      1-7

-------
                                                      OSUER Directive #9028.OOa

1.5       EXPERT SYSTEM COMPONENTS AND ARCHITECTURE

          Expert systems can be viewed as having three unique, isolated
components.   See Section 1.8  for  a graphic representation of  the  components  of
an expert  system.  These are  the  user  interface, knowledge base,  and inference
engine.  Most expert  systems  and  system  development  tools have these
components.   If project managers  and software  developers  are  aware  of  each
component  during the  Definition and Design Phase of  the system life cycle, a
development environment can be selected  that supports the components and level
of complexity desired for  the system.

          1.5.1	User Interface

          The user interface permits the user and system to communicate.  The
user interface may be simple.  It can  consist  of questions posed  to the  user
or optional responses from which  the user makes a selection by typing  in the
response desired; or  menus from which  selections are made using the cursor
keys or a  mouse.  A sophisticated user interface may employ a natural  language
processor  written in  a higher level language to ask  and answer questions,  to
explain or justify the system's conclusions, and to  accept modifications to
the knowledge base.

          1.5.2    Knowledge  Base

          The knowledge base contains the knowledge or expertise of the domain
which is the  designated area  of expertise for  the system.  In many  rule-based
expert systems, the knowledge is  encoded into  sentences using an  "If PREMISE,
then CONCLUSION" format.   [Facts] may  also be  stored in the knowledge  base
using a variety of formats such as STRONG_WIND - TORNADO.  Rules  are used to
encode knowledge that includes reasoning about .the inputs to  the  system,  or
facts already encoded into the knowledge base.  Facts are used to encode
passive knowledge that is  concrete.  For example, each entry  in a table  that
translates temperature readings from Celsius to Fahrenheit could  be included
in a knowledge base as a fact.

          1.5.3	Inference  Engine

          The  inference engine contains  the specific procedures and algorithms
for using  the rules and facts in  the knowledge base  to solve  a problem.   One
possible procedure or strategy used by an inference  engine in a rule-based
expert system is  [backward chaining].  Using backward chaining, the inference
engine first  considers a primary  goal.   The text of  the goal  is matched
against the conclusion.  For  example "if the sky is  cloudy, then  the forecast
might include rain".   In this example  "sky is  cloudy" is  a premise  and "the
forecast might include rain"  is a conclusion.   Rule matching determines
whc:her that  rule will contribute information  to the resolution of  the goal.
If the conclusion of  a rule matches the  goal,  then the premises of  the rule
are considered in turn.  Each of  the premises  is considered to be an
.intermediate  goal.  Results of evaluating each goal  are stored in memory to  be


                                       1-8

-------
                                                     OSUER Directive #9028.OOa
 used when evaluating  subsequent  goals.  This  inference strategy  is riot the
 only inference  strategy.   Refer  to  Section 1.8  for  sources of additional
 information.
 1.6       APPLICATION TYPES

          The ability of expert systems to reason about rules as well as input
 data makes  them particularly adept at handling many different types of
 problems.   This Section describes the types  of problems  and resulting
 applications that are  well suited to the  use of expert systems technology.
 The  generic "types" of expert systems discussed below are well documented in
 the  literature about expert systems.

          1.6.1	Diagnosis and Classification

          Diagnosis and classification expert  systems select  an answer  from  a
 fixed set of alternatives on the basis of information input to the system
 while it is reasoning.   Much notable work has been done  with expert systems  in
 the  field of diagnostics,  where problems  with an object  (animate  or inanimate)
 are  diagnosed from observations.  Medical examinations and  electronic circuit
 board analysis have been successfully emulated by this -type of system.
 Diagnostic systems must be able to handle intermittent symptoms,  causes of
 faults hidden by non-related symptoms, and sometimes inconsistent models of
 complex systems..

          Stanford University's  MYCIN has  performed extremely well in
 diagnosing bacterial infections in humans and 'recommending  antibiotic therapy.
 Another diagnosis and classification expert system, MUDMAN, diagnoses problems
 with "mud" used in NL Baroid's oil well drilling and recommends new
 compositions.

           1.6.2	Data Analysis and Interpretation

          Data analysis and interpretation systems  select a hypothesis  based
 on measurement data and corollary information.  They infer  situation
 descriptions from accumulated sensor data.  Artificial vision, image analysis,
 and  surveillance all employ expert systems,  although work is just beginning in
 these areas.   (Note: this guidance document does not address the more esoteric
 research areas current in artificial intelligence.)  Expert systems can deal
 with incomplete or contradictory data.  In addition, their  explanation
 mechanisms can demonstrate the reasoning that lies behind a complex
 interpretation.  Expert systems are often good at assisting managers in
 managing complex and incompatible data sources in order  to  consistently reach
 a known range of conclusions.

          An example of such a system is  DIPMETER ADVISOR,  developed by
. Schlumberger, which analyzes data from oil well instruments.   Stanford
 University's DENDRAL analyzes molecular structures based on mass  spectrogram


                                       1-9

-------
                                                     OSUER Directive #9028.OOa

data.  Another example of a data analysis and interpretation system,
PROSPECTOR, was developed at Stanford Research Institute to interpret data
about ore-grade deposit sites.

          1.6.3    Design and Synthesis

          Design and synthesis expert systems  configure  objects  such as
computer systems on the basis of a set of alternate possibilities.  The expert
system incorporates constraints that the system must meet as well as guidance
for steps the system must take to meet the user's objectives.  Design expert
systems are used in configuring large mainframe computers,  in designing
circuit boards, and in preparing large annual budgets.  Expert systems permit
the exploration of the consequences of proposed design changes prior to
implementing them.  They also facilitate the subdivision of large problems
into smaller ones and simplify coordination among the subsets.

          Other examples include  Honeywell's SYSCON configures their DPS  90
mainframe computers before assembly at the factory, and Digital Equipment
Corporation's XCON which designs large VAX computers.  HI Class, developed by
Hughes Aircraft, solves circuit board assembly problems.

          1.6.4	Prediction and Simulation

          Prediction and simulation expert systems  forecast what will happen
on the basis of current information .by depending on experience or employing
models or simulation.  Situations involving prediction are well suited to
expert systems.  Crop management and weather forecasting are two areas of
current interest.  Predictor systems must be able to model the ways actions
change over time and to manipulate events that are ordered in time.  They must
also be able to deal with incomplete information, generate multiple scenarios,
and use diverse 'data sources.

          For example,  the USDA's COMAX advises  farmers  on  irrigation,
fertilization, and when to harvest.  Purdue University's GRAIN MARKET ADVISOR
helps farmers determine the best way to market the grain they produce.

          1.6.5    Monitoring

          Monitoring expert  systems obtain data on an ongoing situation
following its predicted or intended progress and alerting the user or system
if there is a departure from the expected or usual.  Expert systems capable of
monitoring their environments compare observations to desired outcomes and
report discrepancies.  They must recognize alarm conditions in real time and
avoid false reports of-problems or emergencies.  Note that  "real time" expert
systems are significantly different than those which allow  the luxury of
reflection.  Because of varying environental factors, monitoring systems must
vary their anticipation of alarm conditions with time and situation.
                                      1-10

-------
                                                     OSWER Directive 09028.OOa

         Air  traffic  control and nuclear power plant management are two
current fields of application.   Additionally,  NASA's NAVEX monitors controls
on space shuttle flights.

         1.6.6	Instruction

         Instruction  expert systems  are built depending on user needs  to give
advice, furnish information,  or perform various subtasks.   Expert systems that
can serve as an on-line tutorial aid to students and diagnose areas of
deficiency involve the automation of instruction.  These systems construct a
hypothetical description of a student's knowledge in order to interpret the
student's behavior.  They identify remedial courses of action and develop
tutorial modes of communicating the remedial action to the student.  They can
be used to tutor novices and to advance the development of human experts.

         For example, DOPPLER  DIAGNOSIS,  developed by  Dr.  Evlin Kinney,  helps
train doctors  in the use of non-invasive echo effect equipment.  CBT ADVISOR,
developed by Courseware, helps determine the suitability of instructional
units for computer based training.

          1.6.7    Planning

          Planning expert systems  select a series of actions from  a complex
set of alternatives to meet a user's goals.  Because time and resource
constraints may not permit all goals to be met, the most desirable outcome is
sought.  Planning systems are used in contingency environments.  A planning
expert system  carries out prepared plans of action and can be used in a time'
critical situations such as responding to a. natural disaster.  Priorities must
be established in order to resolve conflicts among goals and subgoals may need
to be established to  simplify complex interactions.

          IBM's UNIT COMMITMENT ADVISOR assists  in the  shut down of power
plants.  Oak Ridge National Laboratories has experimented with several shells
with hazardous spill  containment.

          1.6.8    Control

          Control is a combination of monitoring a system and taking
appropriate actions.  Control systems interpret, predict, repair,  and monitor
system behavior.  Thus, they incorporate many of the other problem areas
presented here.  Problems addressed by control systems include battle
management, mission control, and business management.

          Hitachi's TRAIN BRAKING  ADVISOR regulates locomotive braking for
accuracy and comfort.  COMPONENT IMPACT ANALYSIS SYSTEM, developed by Argonne
National Laboratories, advises nuclear power plant operators on valve and
switch settings.
                                      1-11

-------
                                                     OSUER Directive #9028,OOa

          1.6.9    Repair

          Debugging and repair systems specify remedial plans  and apply them
in limited areas.  Computer programming is the most obvious area of
application, but medical diagnostic systems also have debugging aspects.
Expert systems could greatly enhance expertise in automotive,  avionics,
network, and computer maintenance.

          Troubleshooting Aid for F6502,  developed by Tektronix,  assists
technicians in the repair of F6502 instruments.  Stone & Webster's PUMP PRO
advises on preventive maintenance of centrifugal pumps.  Honeywell's PRESS
debugs operating systems software.  MIT's Programmer's Apprentice assists in
automatic code generation and in isolation of logical inconsistencies in
programs.


1.7       PROTOTYPING OVERVIEW

          Because prototyping is frequently used in expert  system development,
the concept is introduced here and developed further in the relevant stages.

          .Prototyping is inherently easier with expert systems than with
building most conventional programs.  Building an expert system is an
incremental process; at any time during development, the partially built
system will function to a limited degree.  The scope and capability of the
system will be limited to the depth and breadth of knowledge that has been
applied to the knowledge base.  When the expert system encounters a case about
which it has not been given any knowledge, it simply will not offer advice or
it may offer nonsense advice.  With most conventional programs during the
development stage of the conventional system's life cycle,  if they start to
function and encounter an area where logic has not been coded,- they will
usually end without warning or explanation.  Because partially completed
expert systems will run, developers can demonstrate the system as it is being
developed, thereby soliciting comments, suggestions, and feedback.

          Expert system development lends itself better to  a. detailed
prototyping cycle than to a long design period for several reasons.  First,
experts prefer to analyze a working system rather than one on paper.  Second,
users are more motivated when they see a working system.  Third, experts can
better cite exceptions to rules that pertain to realistic problems.
Exceptions to rules tend to emerge as problems are encountered.  There are
several categories of prototypes, each with its own role in expert system
development.  They are described in Sections 1.7.1 through 1.7.4.

          1.7.1	Proof-of-Concept-Prototype

          The Proof-of-Concept Prototype is a small working system designed to
provide a preliminary feasibility assessment of the emerging solution before
additional resources are allocated.  It is used primarily as a proof-of-


                                      1-12

-------
                                                      OSWER Directive #9028.OOa

 concept  test to  show inherent strengths  and weaknesses of the  concept.   It is
 most  commonly built in the Concept Phase,  and further refined  in  the
 Definition and Design Phase.   This type  of prototype may consist  of
 "disposable code"  or code that is used only to prove that an idea or  concept
 is  feasible.

           1.7.2	Demonstration Prototype

           If the System Concept meets  all  requirements,  the Demonstration
 Prototype is constructed during the Definition and Design Phase.   Its purposes
 are to codify core knowledge and to verify the knowledge representation
 schemes  and control structures.   The Demonstration Prototype is generally
 small and specialized, based on a narrow subset of the overall problem domain.
 If  major revisions are needed either to  the knowledge representation  or
 control  strategies,, they are made at this point.  Should a different  approach
 appear to be more workable, the Demonstration Prototype can be reevaluated.

           1.7.3     Full Prototype

           During  the  Definition and Design Phase,  the Full System Prototype is
 developed to encompass the entire problem domain.   The purpose is to  verify
 the problem domain, the knowledge representation schemes and control
 structures, data requirements, and interfaces.  If the problem is properly
 scoped,  the Full Prototype will be of a manageable size.  The  prototype
, contains most of the core and supporting knowledge and the agreed-upon
 knowledge representation and control structures.  If major revisions  are
 needed to the knowledge base, the knowledge base,  or the control  structures, a
 new approach can be designed following the guidelines in Chapter  5, Definition
 and Design Phase.

           Prototyping is  an exploratory  process.   If at this point the  design
 is  no longer valid, the prototype can be reevaluated based on  the new
 information and redesigned as appropriate.  The efforts that went into the
 development are not lost; they represent the design phase in conventional
 systems.

           1.7.4	Production System

           Finally,  the Production System is built  during the Development
 Stage.  This is the actual system that will go out into the field, including
 all user interfaces and links to external databases.  After complete  testing
 and validation,  any disclaimers about the use and liability of the expert
 system are added to the system.
                                       1-13

-------
                                                     OSUER Directive #9028.OOa

1.8       SOURCES OF ADDITIONAL INFORMATION

          1.8.1     Books

          Harmon, P. and King,  D.   Expert Systems:   AI in Business,   John
Wiley and Sons, New York, NY, 1985.  This beginner's primer on expert systems
does not assume any background in computer science, or AI.   Contains an
excellent bibliography.

          Hayes-Roth, F.,  Waterman,  D.,  and Lenat,  D.   Building Expert
Systems,  Addison-Wesley, Reading, MA, 1983.  Comprises a good description of
the [architecture] of an "ideal" expert system and compares and contrasts an
implementation in several early shells.

          Waterman,  D.   A Guide to Expert Systems,   Addison-Wesley,  Reading,
MA, 1986.  A well-written introduction to ES technology.  Includes an
extensive bibliography of expert systems in various application areas.

          1.8.2	Organizations. Journals, and Magazines

          AI Expert, 500 Howard Street,  San Francisco,  CA 94105.   This is a
popular magazine for people interested in AI.  The articles tend to be in the
vein of BYTE magazine, although some deal with industry trends and
applications.

          American Association of Artificial Intelligence (AAAI),  445 Burgess
Drive, Menlo Park, CA 94205.  Publications include a directory of members, AI
Magazine, and proceedings of yearly conferences.  The magazine offers a mix of
theoretical, practical, and general interest reading.

          Computer Society of the Institute of Electrical and Electronic
Engineers (IEEE), 10662 Los Vaqueros Circle, Los Alamitos,  CA 90720.
Publications include COMPUTER, IEEE Software, and IEEE EXPERT.  IEEE EXPERT
magazine is devoted  to articles on applied AI; articles on AI also appear -in
the other magazines.
                                      1-14

-------
                                              OSWER DIRECTIVE #9028.00a
1.9    QUICK BRIEFING  ON EXPERT SYSTEMS

-------
What is an Expert System?
      Perspectives
      Definitions
      Component parts
      Examples
                                o
                                (02)
                                O)
                                a

-------
      "Expert Systems11 Comprise a Branch
            of "Artificial Intelligence"
AI is a collection of cutting-edge, integrated concepts and methods
 from a variety of fields:

 — Decision Science
 — Computer Science
 — Linguistics
 — Psychology
 ~ Neurology
 - Mechanical and Electrical Engineering

  and many more...

AI is what enables computers to perform tasks that would
normally require an experienced or trained person

-------
  "Ryert Systems" Are A Practical
               of Artificial Intelligence
Expert systems have proven to be practical in areas such as:

—  Medical diagnosis;
~  Seismic analysis and oil prospecting;
—  Chemical analysis;
—  Preventive maintenance;
—  Crisis response; '
—  Sales and negotiations advice;
~  Games; and
—  Engine fault diagnosis.

 The expert systems arena is one of the most commercially
 advanced branches of AI.
                                                        CO

-------
     Some Classic Definitions of
             "Expert Systems"
    11A computer system that emulates human expertise by making
 deductions from given information using the rules of logical
 inference."
    11A system whose goal it is to perform convincingly as an
advisory consultant, exhibiting human expertise in a given domain
with self-explanation of reasoning on demand."
   "A computer program that has built into it the knowledge and
capability that will allow it to operate at the expert's level. It will
usually be able to explain the lines of reasoning that led to their
decisions."
                                                               o
                                                               1
                                                               m
                                                               at
                                                               s

-------
    Conventional vs. Expert Systems
   Conventional

• Unambiguous data
• Strict formats
• No mid-run explanations
• Numeric processing
• Major effort to modify
      Expert

Uncertain/incomplete data
Rules of thumb
Mid-run explanations easy
Symbolic processing
Easier to modify
                                                      s

-------
       Key Components  of  an Expert System
                                         User
                                    Depending on the system,
                                   users can be technicians or
                                   managers, and can be novices
                                   or experts in the domain of
                                   the expert system.
                    Deccribe current problem

                    & known facts; query system
                    for expert advice.
                   Interrogate the user; provide

                   interim and final expert advice;

                   explain underlying reasoning.
                                   User Interface
         The part of the system that allows the user to communicate with the system, typically using English-like sentences or
       commands. The user interface also gives the user control over output types (e.g., graphics or tabular reports).
L
       Knowledge

           Base
       A collection of rules of
     thumb, fact tables, and
     questions that embody the
     procedures employed by an
     expert and the knowledge used
     by the expert to make decisions.
    Inference

      Engine
 The part of the system which
performs deductive reasoning
based upon the contents of
the knowledge base and the
circumstances of the current
problem.
     Programs
 The software that drives
and coordinates the system,
and which performs "utility"
tasks, such as data base
maintenance.

-------
 The Knowledge Engineering Process
Information is extracted from experts
Information (i.e., rules, facts, and procedures) is arranged in a

knowledge base
Information conflicts and voids are identified and resolved
                                                       o
                                                       ni
                                                       m

                                                       b

                                                       £•

-------
            Hardware Environment vs.
Expert System Application Characteristics
     Environment
     • Microcomputer
 Characteristics

• Rapid/Proof-of-Concept
 Prototypes
• Script-based
• Rule-based with
 limited fact tables
• Limited requirement
 for access to external
 data Files
     • Minicomputer

     -or-

     AI Workstation
• > 1000 rules
> Processing speed a consideration
• Resource-intensive processing
• Efficiency of operation required
 (i.e., AI architecture, or garbage collection)
      Mainframe
• Multi-user access
• >1000 rules
• Processing speed a consideration
• Access Required to Large Data Bases
• Resource-intensive processing
                                                                            §
                                                                            &

-------
                                                     OSUER Directive #9028.OOa
                                  CHAPTER 2
                  IMPLICATIONS  OF USING EXPERT SYSTEMS AT EPA
2.0       PURPOSE OF THIS CHAPTER

          This chapter discusses some policy issues for expert systems.

2.1       EXPERT SYSTEMS FOR POLICY INTERPRETATION

          Expert systems for policy interpretation are not cost-effective for
OSWER because of the dynamic nature of policy.  The maintenance costs of
expert systems  that address policy can be significant due to changing
directives from Congress and regulatory policies.  Projects that address
policy should have sufficiently  short payback to offset their risk of
obsolescence.   Because  of these  difficulties, expert systems may be used to
cite guidance and regulations but policy interpretation should be avoided.
This does not impose restrictions on text retrieval systems -- such as
hypertext and smart  indexing --  but  interpreting the policy is to be left to
human decision  makers.
2.2       LEGAL ISSUES

          Legal issues for expert systems involve all matters of the law,
including liability.  Legal  issues include licensing and distribution rights
to the  software, copyright infringement  risks  for software and documentation,
and  access  to  information used  in the expert systems.

          2.2.1 Licens ing Agreements

          All OSWER expert systems will adhere to the terms of negotiated
licensing agreements  set  up  with software vendors.

          2.2.2 Confidential Information

          Expert systems and their contents are open to public inspection.
Therefore expert systems  should not contain confidential business  information
(CBI) or enforcement  information, nor should they contain data covered by the
Privacy Act.   Information used  in the expert systems will be  made  available  to
the  public  in  accordance  with the Freedom of Information Act.

          2.2.3 Liability

          Liability in the realm of expert systems has received much attention
from computer  scientists, but has not been tested in the courts.   Some
software liability  cases  have been tried -- such as the LOTUS 1-2-3
spreadsheet error suit  -- but the field  is relatively open to interpretation.
                                      2-1

-------
                                           OSUER Directive #9028.OOa

2.2.3.1 System Profile

Expert systems appear to be especially vulnerable to liability
issues because of their advisory nature in judgmental matters and
their high profile.  The high profile can be addressed by
classifying the use of an expert system as an assistant or advisor.
This sets the user's expectations to a more reasonable level.  If
people believe that they are receiving accurate, "expert"
information, then* they may act on it without the proper skepticism.

2.2.3.2 User Registration

Another means of reducing liability is to control the people who use
the expert system by registering, or characterizing and qualifying
them.  Registering the users of expert system is discussed in
Chapter 7. Characterizing and qualifying users involves three steps.


First, developers must assess the degree of computer literacy and
competency in the subject area of the intended users and build the
system accordingly.  Second, the users must be made aware of the
level of complexity and scope of the expert system via disclaimers
and documentation.  Finally, initial and on-going training is
essential, especially for first-time users of the expert system.
User qualification can be enforced by requiring that the user
specify a registration number for the expert system when requesting
user or technical support.                     .—  -

2.2.3.3 Disclaimers

The proper use of disclaimers (Fig. 2.1) will also reduce the risk
of liability for expert systems.  Disclaimers should include
information on:                                        •

o         The characteristics of the intended user of the expert
          system;

o         The scope and applicability of the system;

o         Assumptions made in the development process;  and

o         Intended uses for the system.

It should be explicitly stated that using the expert system  for
purposes other than  those intended will produce unpredictable
results, and that neither EPA nor its  contractors are responsible
for  the consequences resulting from intentional abuse or
unauthorized use of  the system..
                             2-2

-------
                                               OSWER DIRECTIVE 0902B.OOE
                     ., an expert system that
                           B        task is
automates the 	
to be used for the purposes of:.
	5___, and	
     Recommendations presented by the
     system should always be evaluated
     by the user for content and applicability
     to the problem. Users will be held
     responsible for any and all actions based
     on the system's recommendations.
                      Kev

A  - name of expert system
B  -title for task automated  by expert system
C, D, E  - activities automated  by  the expert  system
                   Figure  2.1
 Example of  an  Expert  System  Disclaimer

-------
                                           OSUER Directive #9028.OOa

2.2.3.4 Liability Scenarios

For now, EPA should consider expert systems to be  tools,  with the
decision-maker retaining the responsibility for  any  liability
resulting from the decision.  EPA managers  are not divested of
liability, but can reduce it by emphasizing the  advisory nature of
expert systems.  Some researchers suggest that expert systems are
simply automated information retrieval systems and should be held no
more liable than publishers of books.  Another point of view
suggests that expert systems be held liable as independent experts,
and thus are fully responsible for all recommendations.   In other
words, the expert system is just a tool that people  may use at their
discretion, but may in no way hold responsible for the outcome.

Finally, there is the possibility that people may be held liable for
not using an expert system.  If a mistake is made that could have
been avoided by using an expert system, the decision-maker might be
found at fault.  Under each of these liability scenarios it is
important to think about the extent of liability for each person
involved - - including the expert, the developers, the users, and the
maintenance staff.

2.2.3.5 Private Industry

Developers of EPA systems should consider  the liability  implications
of EPA-sanctioned software being used by private industry.  Expert
systems developed by EPA can have broad ranging effects  in the hands
of unauthorized users.  Thus EPA should implement a user
qualification system and a. user registration system  (see CHAPTER 7)
to mitigate these effects.  Additionally,  EPA expert  systems may
wind up in the hands of the regulated community as a  result of a
Freedom of Information Act  (FOIA) request.

2.2.3.6 Distribution

Because of the potentially  extensive user  base  of each  expert
system, it is necessary to  consider  the implications  of their use on
Regions, states, and industry.  Three mechanisms for  distributing
expert systems are:

o         User registration,

o         Mailing lists,  and

o         Open dissemination.

The extent of risk for misuse  of an  expert system increases the more
freely it is distributed.   The type  of distribution is  left to the
project manager's judgment based on  the  sensitivity of  the
information "contained in the expert  system.

                            2-3

-------
                                                     OSWER Directive *9028.00a
2.3       ORGANIZATION'S CULTURE

          The initial approval to proceed with an expert system project and
its final acceptance depend upon an organization's culture.  In addition, the
culture will in some way be affected by expert systems technology, if it is
incorporated into the operation of the organization.

          2.3.1 Level of Acceptance

          The level  of acceptance of new technology depends on an
organization's responsibility and risk tolerance.  Conservative organizations
that operate in a well-defined environment are often reluctant to try advanced
technical information system solutions.  Other organizations view advanced
technology as an opportunity to increase productivity and capabilities, and
thus welcome expert systems.

          2.3.2 Developer Initiative

          Developer  initiative is heavily influenced by an organization's
culture.  Organizations that encourage individual initiative will stimulate
more activity in new technologies, and are therefore more likely to develop
grass-roots expert system ideas and projects.

          2.3.3 User Acceptance

          User acceptance also depends on the nature of the organization.   The
successful introduction of new technology is often based on how the users of
that technology are received.  In other words, if users are encouraged and
rewarded for their efforts, the hew technology is likely to succeed.

          Another factor that affects user acceptance is management
commitment.  Positive reinforcement from management encourages user
acceptance.  Involving the users in the early stages of the expert system's
development increases their feeling of ownership and thus promotes user
acceptance.  If the users are indirectly discouraged or penalized for aborted
attempts, the new technology often remains unused.  This is especially
critical during the prototyping stages.

          2.3.4 Prototyping

          Organizations that embrace new technology should understand the
exploratory nature of rapid prototyping used in developing expert systems.
Rapid prototyping provides the flexibility to make false starts within a
project  (see Section 1.7).  Because prototypes are not all-or-nothing efforts,
they may be discarded in favor of a new approach as new information is
discovered.  The success of expert systems depends on organizations being
tolerant of the exploratory process that might not have immediate benefits.
                                      2-4

-------
                                                     OSWER Directive 09028.OOa

          2.3.5 Effects of Expert Systems

          Changes in the operation of an organization lead to changes within
the organization's culture.  Expert systems can cause changes in two ways.
First, in building an expert system to automate a task, the task itself comes
under scrutiny.  This in itself often causes changes in the procedure.
Second, expert systems affect the information task itself by changing the
user's input routine, the time involved in finishing the task, the answer and
justification received, and potentially in other quantitative and qualitative
ways.

          Two other facets of expert system development that are influenced by
an organization's culture are effects on expert system development staff and
ramifications of early expert system projects.  Finally, the organization's
culture determines the amount of emphasis that is placed on early expert
system initiatives.  Further efforts in expert systems may hinge upon the
success of earlier ones.
                                      2-5

-------
                                                     OSWER Directive #9028.OOa

2.4       CROSS CUTTING CONSIDERATIONS

          Several topics of particular importance are addressed in multiple
phases of the system life cycle.  These topics include:

          o        Project Management Flan;
          o        Project Participation;
          o        Project Reviews and Quality Assurance;
          o        Project Approvals;
          o        Methodologies  and Tools;
          o        Benefit-Cost Analyses; and
          o        Knowledge Management.

          This Section briefly describes  each topic from a cross cutting,  life
cycle wide perspective.  Specific activities relating to each of these topics
are presented in the remaining chapters of this practice paper.

          2.4.1  Project Management Plan

          The Project Management Flan is  the fundamental document for planning
and managing the system life cycle, and is mandatory for every project.  It is
first developed in the Initiation Phase and is updated, expanded, and refined
continually throughout the life cycle.  Refer to the OSWER Project Management
Practice Paper for additional information.

          Several important implications  of expert systems for the Project
          Management  Flan are noted below:

          o        Explicit determination of the relevance of the application
                   area and type  must be documented  in  the Project Management
                   Plan.

          o        Although prototyping can be an  iterative, repetitive
                   process, justification for each successive prototype must
                   be  documented  in the Project Management Plan.  The       ;
                   "lessons learned" in each iteration  should be spelled out.

          o        Any modification to the standard  life  cycle  stages and
                   phases to accommodate prototyping or other aspects of
                   expert system  development must  be documented in the
                   Project Management Plan.

          o        Knowledge base testing, validation,  and maintenance plans
                   must also be spelled out in the initial plan and  refined
                   over time.
                                      2-6

-------
                                                     OSUER Directive *9028.00a

          2.4.2  Project Participation

          Information management problems and systems to address them require
the participation of organizations and individuals with a diverse set of
experience and skills.  These range from an in-depth understanding of
pertinent agency programs to expertise in specific information management
technologies.  Successful projects require that certain important roles be
specifically assigned to organizations and individuals.

          Expert system development efforts will require the roles common to
all system development  projects.  These are OSWER Program Management, OSWER
Program Execution, Project Management, Project Execution, Quality Assurance,
and Procurement.  Within Project Execution there are the following specialized
roles.

          o         [Domain Expert]  -  the expert provides the information or
                    expertise for the  expert  system.  The expert acts as  the
                    functional expert;  providing expertise,  defining  the
                    boundaries of the  domain,  and providing  validation for the
                    expert  system.   The expert's qualification is to  have in-
                    depth knowledge of a functional  area.

          o         [Knowledge Engineer]  - .the knowledge engineer elicits the
                    information or expertise  from the expert.  The knowledge
                    engineer's role in an expert system project is to gather
                    the information or expertise of  'the expert and to codify
                    that information using expert system techniques.   The
                    knowledge engineer's qualifications are  to be familiar
                    with expert system development tools and methodologies.

          o     •    User -  the user is a person who  requires the functional
                    expertise of the expert on a regular basis.   This
                   . individual, over time, will become a functional expert.
                    The user's role in an expert  system project is to use the
                    system to perform  tasks that would normally require input
                    from an expert.  The user's qualifications are to have a
                    willingness to work with new systems and to understand the
                    functional area to the degree  that he  or she can identify
                    when the expert system is providing incorrect
                    recommendations.

          2.4.3	Project Reviews and Quality Assurance

          Independent review of the products of the system  life  cycle is
performed to  ensure that the project  team .is  proceeding in  an appropriate
direction to  effectively solve the  information management problem.   The   ,
reviews  address  programmatic issues,  technical considerations,  and project
management.   The reviews provide feedback to  the project team as well as
advice to the individuals  required to approve the project.   Project  reviews


                                      2-7

-------
                                                     OSUER Directive #9028.00a

and other quality assurance activities are performed throughout the life
cycle.

          Several important aspects of expert system project reviews and
quality assurance are noted below:

          o        The reviews to be conducted and other planned quality
                   assurance activities are documented in the Project
                   Management Plan.

          o        Quality assurance applies to all aspects of the project,
                   and quality assurance is a continual part of the project
                   effort.

          o        Formal reviews are structured to ensure a level of review
                   commensurate with the nature and scope of the information
                   management problem and potential solution.

          o        The results of reviews are included in each decision paper
                   developed at the end of each stage of the life cycle
                   management process.

          o        Knowledge base testing and validation will require
                   separate verification of data (facts) and logic (rules)
          o        Detailed test cases and review and approval of the system
                   by the expert will be required.

          o    •    Problems particular to knowledge bases will have to be
                   isolated and corrected such as subsumed rules,
                   contradictory rules, unreachable goal states.

          2.4.4	Project Approvals

          Formal  approvals  are provided  throughout  the  system life  cycle to
ensure that OSWER management supports the project and is in agreement with the
chosen project direction.  These approvals are provided at a level
commensurate with the nature and scope of the system.  Review and approval
thresholds established for conventional.systems may not be appropriate for
expert systems.  The following is a review structure that considers impact,
risk, level of interest, usage distribution, development effort, and system
type follows:

          o        Impact can be measured in several different ways.  One  is
                   to estimate savings in terms of some combination of
                   professional labor hours saved, support labor hours saved,
                   computer time saved, or reduced contractor assistance.

          o        Risk  is a measure of a system's susceptibility to
                   incorrect or partially incorrect results and the probable
                   significance of such results.  The measurement of risk

                                      2-8

-------
                                                     OSUER Directive *9028.00«

                   takes into account such factors as the cost of incorrect
                   or partially incorrect results, a cost-benefit analysis of
                   the system, and a risk analysis of the "regret factor."

         o         Level of interest in a particular system is a measure of
                   its visibility.  Exposure of the system within the agency,
                   the government as a whole, or the public eye, will
                   influence the degree of review and approval required for
                   the system.

         o         Usage distribution will affect review and approval
                   requirements.  One possible distribution categorization is
                   expert systems developed for individual use systems, work
                   unit distribution, Agency-wide distribution, and
                   distribution beyond the Agency.  Each level of
                   distribution would require increasing levels of review and
                   approval.

         o         Development effort is a measure of the amount of resources
                   invested in the development and maintenance of an expert
                   system.  Investments should be calculated in terms of
                   dollars and labor hours (usually full time equivalents)
                   required to develop the expert system.

         o         System type or application domain of a given expert system
                   will also  affect its level of review and approval.  Some
                   domains such as hazardous waste control will demand near
                   perfection and resulting  intensive review.  Other domains
                   such automated forms processing may require much less
                   review.

         2.4.5    Methodologies and Tools

         Expert systems projects  will clearly use different methodologies and
tools than those used in conventional systems development.  Several important
expert system considerations  in the use of methodologies and tools are noted
below:

         o         While the  LCM Guidance does not mandate the use any
                   specific methods or automated tools, it does require that
                   choices be made explicitly.  Proper selection of
                   application types and areas facilitate identifying
                   required expert system shell characteristics, which in
                   turn lead  to specific products.

         o         Successive prototypes must be justified individually and
                   must be documented in the Project Management Plan.

         o         Methodologies and tools should be considered from a full
                   system life cycle perspective.

                                      2-9

-------
                                                     OSUER Directive #9028.OOa
          o        The methodologies and tools for each phase or stage are
                   selected no later than the end of the preceding phase or
                   stage.

          2.4.6    Benefit-Cost Analyses

          The important decisions  of the system  life  cycle  and the prescribed
management approvals often hinge on two key questions:  "How much will it
cost?", and "Are the benefits worth the cost?"  Benefit-cost analyses are
particularly important in expert system development because of the unique
hardware, software, and developer skills that may be involved.  However,
quantifying these measures for expert systems can be difficult, so alternate
measurement techniques should be considered.  These may include:

          o        Lost opportunity cost analysis,
          o        Relative worth methods,
          o        Quantitative bounding, and
          o        Qualitative bounding.

          2.4.7	Knowledge Management

          Knowledge management  is  the process  of formally controlling the
initial development of a knowledge base and all subsequent changes or
additions to it.  It ensures that the integrity of an expert system is
maintained throughout its life cycle.  Knowledge management is closely related
to configuration management.  Refer to the OSWER Configuration Management
Practice Paper for a more complete description.

          There are several advantages  of applied knowledge management over
less structured knowledge base development.   As the agency develops multiple
expert systems over a period of time, opportunities will present themselves to
gain leverage from previous efforts.  Well-structured knowledge bases will
permit the development of reusable modules of knowledge stored in a "knowledge
library."  A reusable "knowledge dictionary" will enhance expert system
development by reducing the time and cost required to specify commonly used
knowledge -base elements.
                               «
          As OSWER increases its  expert system development  activities, it
should make a deliberate effort to seek out other systems currently under
development in order to identify opportunities to share knowledge.  If
multiple applications can access common knowledge bases, they can be cost-
justified more easily and accurately.

          Linking modules  together to form a knowledge base may impact system
performance adversely or positively.  On the one hand, the control structure
required to interface the different modules will consume some portion of the
processing capabilities of the system.  On the other, breaking a knowledge
base into modules may permit loading and running only those modules needed for
the current consultation.

                                     2-10

-------
                                                     OSUER Directive #9028.OOa


         Knowledge  management  is  an ongoing process  because expertise is
constantly evolving and changing.   Proposed changes and additions to the
knowledge base must undergo formal review before they are made to ensure the
integrity of the system.  Some requested changes may be of such significance
that they must be addressed through a separate iteration of the life cycle.
Requests which will have the effect of changing the types of data and/or
knowledge to be processed by the system generally should be handled  in this
manner.
2.5       RESOURCE REQUIREMENTS

          Resource requirement considerations comprise a significant portion
of the planning process for expert system development.  Particular attention
should be paid to  scarce resources such as an expert's  time.  Expected
requirements for resources in  each phase of.  the life  cycle are discussed
below.

          2.5.1  Experts

          The single most important resource in the development of an expert
system is the time of  the domain experts.  Their  time is  also a scarce
resource because of the demands that placed  on them by their organization.
The commitment of  an expert's  time -is  a plus in terms of  management  support,
but it is also a risk  in terms of not  wasting this business resource on
fruitless efforts.

          2.5.1.1 Initiation Phase

          The experts are needed as a resource in the Initiation Phase  to
          assist in determining the scope of the system because they know the
          problem area and solution requirements best.

          2.5.1.2 Concept Phase

          Experts are needed in the Concept  Phase to  determine the initial
          expert system functions and  features.  In addition, the experts will
          develop a preliminary overview of  the problem that allows  the
          developer to build a good proof-of-concept  prototype.  Involvement
          of the experts is extensive  in the Concept  Phase.

          2.5.1.3 Definition and Design Phase

          The experts should be prepared for a significant time investment in
          the Definition and Design Phase.   During this phase the experts must
          explain  the problem  to knowledge engineers  and  to management.  The
          experts  are also crucial for the development of the demonstration
          prototype of the expert system built in this phase.


                                      2-11

-------
                                                     OSUER Directive 09028.OOa

          2.5.1.4 Development Stage

          When determining the amount of time  needed from the experts,
          remember the experts are going to  be needed more in the  initial
          startup of development where most  of the knowledge acquisition takes
          place.   As the development progresses,  the experts will  need to be
          consulted on a regular basis.   The experts will also be  needed
          during testing,  verification and validation.

          2.5.1.5 Implementation Stage

          During implementation of the system,  experts will be needed to a
          lesser extent than during the Development Stage.   They will be
          needed mainly to resolve any problems that arise during  the beta
          test.   It may be useful to find another expert in the field who was
          not involved in the development of the system to serve as  a beta
          tester.  The additional expert will  be able to test the  system more
          rigorously than other beta users,  and-could assist in a  final
          validation of the system.

          2.5.1.6 Operation Phase

          Experts will be needed to assist in  modifying the knowledge base in
          the Evaluation Stage.  They will also be helpful in determining the
          degree to which the system continues to address the information
          management problem.  The size of the system and the volatility of
          its problem domain will dictate how  much time will be required from
          the experts.

          Experts may also be helpful in determining what portions (if any) of
          the terminated system to retain in the Archive Stage.

          2.5.2 Information about the Problem

          Information about the problem will be required to verify that the •
problem exists and to justify  the  system development effort to solve it.  The
information will be gathered in the Initiation Phase and will be used to
document the existence- of the  problem and, when available, describe solutions
to similar problems.

          2.5.3 Knowledge Engineers

          Knowledge engineers perform three  major functions in the expert
system development life cycle.  First,  they help  structure the problem,
perform feasibility studies, and design the expert  system.  Second, knowledge
engineers extract knowledge  about  the problem  solving process  from the
experts.  Finally, .they are  responsible for knowledge management  in the  expert
system.
                                     2-12

-------
                                                     OSUER Directive #9028.OOa

          Because of their important role  in the  development process,
knowledge engineers are extensively involved in the expert system project from
the Concept Phase on.

          2.5.3.1 Concept Phase

          Knowledge engineers perform a significant portion of their work in
          the Concept Phase with responsibilities including:

          o         Selecting the knowledge  representation  and control
                    structure,

          o         Developing the Knowledge Management  Flan and System Test
                    Plan,

          o         Helping determine the  functional requirements  of the
                    expert system,

          o         Assisting in performing  a feasibility assessment for the
                    expert system approach,  and

          o         Building the Proof-of-Concept prototype.

          The amount of time spent by the  knowledge engineers will vary with
          the size and complexity of the application.

          2.5.3.2 Definition and Design Phase

          The number of knowledge engineers  is dependent upon the estimated
          size and complexity of the problem.  It is recommended that  two
          knowledge engineers interact with  the experts in order to provide
          different points of view in the knowledge acquisition process.
          During design, the role of the knowledge engineers is to assist in
          identifying potential knowledge sources and to help the experts to
          explore possible avenues in the problem domain.  However, it is
          important for the knowledge engineers to help the expert scope the
          problem domain down to a reasonable size in order to design  a
          technically feasible expert system.

          2.5.3.3 Development Stage

          The knowledge engineers will be needed throughout the Development
          Stage.  The project schedule hinges on their  level of commitment.
          Initially, the knowledge engineers will be involved extensively in
          documenting the expert's knowledge.  The knowledge engineers also
          provide support to the programmers; in some cases,  the knowledge
          engineers will perform both knowledge engineering and programming.
          Finally, the knowledge engineers will interact with the experts,
          programmers (if used) and management during the  testing and
          validation phases of development.

                                      2-13

-------
                                                     OSUER Directive #9028.OOa
          2.5.3.4 Implementation Stage

          Knowledge engineers will play a much lesser role during this stage
          than the Development Stage.   They will be needed mainly for
          contacting the experts if any problems arise during beta testing.
          They will also be involved in analyzing the comments and suggestions
          of the beta users,  and making any necessary revisions to the system.
          Knowledge engineers may also be involved in the training process and
          will provide expertise for the expert system documentation.

          2.5.4 Programmers

          Programmers are responsible for the development of software
associated with the expert system.  This includes placing the knowledge
acquired from the experts into the development environment, writing
interfaces, developing support programs, and integrating the expert system
into the existing information processing architecture.  In some projects, the
knowledge engineers may serve as the programmers.

          2.5.4.1 Concept Phase

          Programmers who have experience with expert systems are necessary to
          help determine the testing methodology in the Concept Phase.  They
          also assist the knowledge engineers build the Proof-of-Concept
          prototype.

          2.5.4.2 Definition and Design Phase

          It is the responsibility of the programmers to incorporate the data
          extracted during knowledge acquisition into the specified
          development environment.   They can be used to assist in the
          requirements definition process and as a technical resource for
          programming issues that need to be addressed in the design.  They
          are also useful for building the Demonstration Prototype during this
          phase.                                              •

          2.5.4.3 Development Stage

          In the Development Stage, the programmers are responsible for
          helping codify the experts' knowledge and for writing support
          programs. They will be the key people in building the knowledge
          base, developing the interfaces, and responding to the users
          suggestions on how to improve the expert system.  Their services
          will continue through testing and validation where they will be
          responsible for any changes necessary before the Implementation
          Stage.
                                     2-14

-------
                                                     OSWER Directive #9028.OOa

         2.5.4.4 Implementation Stage

         The programmers will be  involved  to  a  lesser  extent  in this stage
         than  in the Development  Stage.  They will be  involved in making any
         necessary  revisions to the system following the beta test,  and may
         provide some  input for the documentation on the system.   Programmers
         could also be involved in technical  support to the beta  users,  if
         there are  any problems at the program  level or problems  with
         hardware and  software compatibility.

         2.5.5 Data Collection

         Effective  data collection facilitates  the development of expert
systems.  Most of the data collection is performed in the early stages of the
life cycle.

         2.5.5.1 Concent Phase

         Initial data  collection  functions are  limited to  those necessary in
         determining the conceptual model's features and inputs,  user
         training,  system benefits and  costs, end users, and  system scope.
         Potential  sources of  information  include other expert systems that
         are  already built,  in the Initiation Phase, or under development.
         Resources  are mostly  those of  the knowledge engineer's time.

         2.5.5.2 Definition and Design  Stage

         Further data  collection  required  during the Definition and Design
         Phase involves information about  the problem  domain, the development
         environment,  and the  targeted  delivery environment.   To  properly
         define and scope the  problem domain, possible data sources should be
         identified and realistic objectives should  be established.

         In the selection of the  development and delivery  environments,  much
         information is required  in order  to make a  knowledgeable decision.
         For  example,  in order to assess the cost of the system,  the hardware
         requirements  should be determined for  both  development and delivery
         environments.

         2.5.5.3 Development  Stage

         In the Development Stage, data will be collected  on  user feedback,
         management feedback,  and the development process.  Also, any data in
         the  form of spreadsheet  and  database will need to.be documented in
        . the  development plan.  Use the  data management plan  as a guide in
         the  data collection process.
                                     2-15

-------
                                                     OSUER Directive #9028.OOa

          2.5.6 Licensing Fees

          It is important to consider  licensing fees for both the development
and the delivery environments when estimating the resource requirements for
the expert systems.  The licensing fees constrain the number of copies of
software that may be used in developing the expert system, and also the number
of copies of the expert system that can be distributed.

          2.5.6.1 Concept Phase

          Planning for software  runtime licensing fees  is necessary in this
          phase because both the costs to  build and to  field the expert system
          are based on the number and  location of the users.

          2.5.6.2 Implementation Stage

          As the system is nearing distribution to field users,  a staff member
          will be needed to check all  licensing and run-time fees with the
          company who distributes the  shell.   All fees  must be paid,  contracts
          and agreements must be activated,  and any other legal  matters must
          be addressed at this time.

          2.5.7 Testing

          The testing methodology needs to be set in the Concept Phase.  Both
conventional software and expert system testing methodologies should be
considered.  Testing is done by the development staff if performed during the
Development Stage.  The testing includes validation of the data, knowledge
base, and logic flow.

          During the Implementation Stage, the system will be tested in the
field by potential end users.  Comments received from those testing the
product will be incorporated into the final revisions.   This testing will help
to ensure user acceptance of the system.

          2.5.8 Maintenance

          Maintenance plans should be  started in the Concept Phase.  Questions
such as to who will be responsible for maintenance, when it will be performed,
how volatile and dynamic the problem domain is, and what role the users will
play need to be addressed.  A staff member or contractor will be needed to
continue to maintain the expert system, as well as provide technical  support
to the beta users during the Implementation Stage.

          Maintenance is necessary in the  Operation Phase to ensure that any
previously undetected  errors can be corrected.  Maintenance also provides  a
means to take  advantage of  hardware upgrades, new releases of system  software,
arid applications packages  that  enhance  the system's performance.
                                     2-16

-------
                                                     OSUER Directive #9028.OOa

          2.5.9 Hardware

          The development hardware will be needed from the Concept Phase
through the Development Stage.  For more information on selecting suitable
hardware refer to Section 5.4.1.3.  The beta users will provide their own
hardware during the Implementation Stage.

          Changes to the expert system, prompted by advances in hardware and
software, may be necessary during the  Operation Phase.  Changing the hardware
platform of an existing system should  be avoided  if at all possible, but if
the existing platform  is no longer supported or becomes obsolete, it may be
unavoidable.  Such a change will likely warrant a separate iteration of the
system life cycle.

          2.5.10 Software

          By the end of the Concept Phase an. appropriate development vehicle,
including the expert system shell, environment, and language, has been
selected. The key issues here are how  to encode the knowledge and how to
integrate the software packages to expedite development.  Topics to consider
are:

          o         Interfaces between  any external  and/or utility  programs.
                    Also,  any  links between quantitative  techniques  and  the
                    expert system  should be addressed.

          o  .       Identify and use  existing  software development  aides such
                    as  intelligent editors, debuggers  and source code
                    managers.

          Copies of the run-time version of the system will be made during the
Implementation  Stage and sent to  the beta  users.  Over the  span of  the
Production  Stage, the  knowledge base will  likely  be modified one or more
times.  These modifications may entail interfacing  the expert system with
newly developed systems  or upgrading the  system's shell.  All software  and
data that are unique and essential to  the  system's  operation should be
archived  for possible  future use.
                                      2-17

-------
                                                     OSUER Directive 09028.OOa
                                  CHAPTER 3
                               INITIATION PHASE
3.0       INTRODUCTION

          Initiation is  the  first  of  five major phases  in the OSWER system
life cycle.  The other phases are Concept,  Definition and Design, Development
and Implementation, and'Operation.  This phase provides the first formal
description of the information management problem and secures the resources
needed to examine the problem and potential solutions.   The most significant
activities of the Initiation Phase include:

o         Confirming the existence of a problem,  providing additional guidance
          where  appropriate,  and approving  (or rejecting) the commitment of
          resources to begin the Concept Phase.  This activity is performed by
          program management.

o         Preparing an Initiation Decision  Paper  that identifies and describes
          the information management  problem and  brings it to the attention of
          OSWER  program management.  This activity is performed by one or more
          program organizations.

o         Preparing a Project Management Plan to  describe the initial approach
          for managing and conducting this  phase  and future actions under the
          remainder of the life cycle.  At  this point,  the Project Management
          Plan provides  detail 'for the Concept Phase; it is revised and
          enhanced in later phases.

          Clearly identifying and describing the  information management
problem is critical to the successful development of an  appropriate solution.
How the problem  is defined during  Initiation will shape  the.analyses and
decisions of the subsequent life cycle phases.

                   During the  Initiation Phase,  there'should  be no
                   assumption  that the  solution  will necessarily
                   be an expert system!

          Even in cases  where the system developers have preconceived notions
that an expert system is  the preferred solution,  the analysis in the
Initiation Phase should not be  intentionally slanted to  use expert systems.
Instead, the primary goal of this  phase is to  describe the information
management problem objectively, independent of potential  technological
solutions.
                                      3-1

-------
                System Us Cycto
                      for
                 Export Systems
                                                                         OSWER DIRECTIVE #3028 05s
Anpfarantrtfan
                Objectives
                 Decisions
                 New
                 Products
To  describe  the  problem  In  clear,  technology-Independent
terms upon which  organizations  can agree

To  determine whether staff  or other  resources  will be
devoted to  defining and  evaluating alternative ways
to respond to the  Identified  problem In  the Concept  Phase
    Project  approach  decisions  Including:
    • who  will  manage the  project
    • who  will  participate  In  development
    - what  reviews  are necessary (and  by whom)
    - what  approvals  are  necessary  (and  by  whom)

    Execution decisions Including:
    • what  Is  the  problem
    - what  organizations  are-   involved
    • what are the  sources  of  data,  the  scope of the
      solution,  and  the  Information  requirements
      such  as  report generation
 •  Initiation  Decision Paper

 •  Project Management  Plan
                                             Figure  3.1
                                          Initiation  Phase
                                Objectives,  Decisions, and  Products

-------
                                                     OSUER Directive «9028.00a

3.1       OBJECTIVES

          Figure 3.1 graphically illustrates the objectives associated with
the Initiation Phase.

          The primary objective of the Initiation Phase is  to describe the
problem in clear, technology-independent terms upon which all pertinent
organizations can agree.  If an expert system may be the technology chosen to
resolve the problem, then the project manager and software developer should
consider the expert system application success factors listed in Section 3.3
before completing this phase.

          The second objective  for the Initiation Phase is  to determine
whether staff or other resources will be devoted to defining and evaluating
alternative ways to respond to the identified problem in the Concept Phase.
Committing resources beyond the Concept Phase is premature at this point.  The
use of rapid prototyping as a tool to work through these issues is a feasible
approach at this stage and in the Concept Phase.


3.2       DECISIONS

          There are  three  types of decisions made in the Initiation Phase:
project approach, execution, and continuation. . Figure 3.1 graphically
illustrates these decisions.

          3.2.1    Project Approach

          The project approach  decisions  address the organization of the
project and the participants in the project activities such as system
acceptance testing,  reviews, and approvals.

          Decisions  are focused on project management and controls,
establishing:       .

          o        Who will manage the project,
          o        Who will participate  in  development,
          o        What reviews are necessary  (and by whom), and
          o        What approvals are necessary  (and by whom).

          3.2.2    Execution

          The execution decisions address the scope and specific features of
the system.  These decisions address programmatic, technical, and system
support related issues.

          Programmatic issues include deciding what the problem is,  whether an
organization is currently responsible for solving the problem, and which
organization(s) should receive the proposed automated solution.  In resolving


                                      3-2

-------
                                                     OSUER Directive #9028.OOa

these issues the emphasis in the Initiation Phase should not be on a special
technology (e.g., optical disk technology,  expert systems,  etc.)  but should be
technology-independent.

          3.23	Continuation

          The continuation decision confirms  that a defined information
management problem exists and is significant enough to warrant further
investigation.   The decision confirms that the information management problem
is beyond the capabilities of existing systems and that developing a new
system has merit.  The decision made here also has to do with resources, and
this is an assessment of return on investment, and that is that the investment
of people and dollars will produce results which are desired by the
organization in terms of productivity.

          Technical issues include determining sources of data,  the scope of
the solution, and the information requirements such as report generation.
When there is a mix of conventional systems' and expert systems -- called
hybrids or embedded systems, new information needs may be associated with the
conventional systems being considered; in this case, the Initiation Decision
Paper should identify the new information needs.  A systems support issue is
to determine who will be responsible for the system once it is fully
operational.


3.3       SUCCESS FACTORS

          Several factors that can impede success if not considered in the
Initiation Phase are described below.  This Section focuses on perceptions  of
what expert  systems can  and cannot do, and it offers solutions or
alternatives.

          3,3.1	System Development  Bias

          Because expert systems are a relatively new technology, individuals
and groups  involved in initiating  a new system may have little or no
familiarity with them.   They may even have a negative  impression of their
capabilities.  On  the other hand,  some parties may  tend to  find an expert
system solution  to every problem, whether it is  appropriate or not.  Before
undertaking  an expert system project,  it is important  to determine that the
problem has  not  already  been solved by other types  of  conventional programming
such as modeling,  decision support, databases, or  linear programming.

          3.3.2     Scoping the  Problem

          Expert system development projects must be reasonably scoped from
initiation  onward.  Failure to do  so can lead to developing a solution to the
wrong problem or tackling  overly complex or nebulous problems.
                                      3-3

-------
                                                     OSWER Directive #9028.OOa

          3.3.3  The Problem is Well Suited to ES Technology

          The key characteristic of an application that will help the project
manager or software developer determine whether expert system technology
should be used is the type of problem.  If the problem is purely algorithmic
or procedural in nature, then it can be addressed by conventional technologies
more efficiently than by expert systems.  If the type of problem requires
symbolic reasoning  (as described in CHAPTER 1, Expert Systems Defined, then
the problem may be  suitable for expert systems technology.

          3.3.4  An Expert  is Available

          Expert systems are developed by taking the specific knowledge of an
established expert  and putting it into a system.  Some systems may incorporate
the knowledge of more than one expert, while others reflect the knowledge and
strategies of a single individual.  Finding the right expert(s) is a key step
in building an expert system.  If no true expert exists, then the problem may
be too nebulous and ill-defined to be effectively addressed by an expert
system.

          3.3.5  The Problem Domain has Bounds

          Once an expert is identified,  the project manager or software
developer should consider whether the set of solutions to the problem has
boundaries or whether there are infinite solutions.  The problem cannot have
an infinite set of  solutions.  The solutions that will be considered by the
system must be determined in advance.  When the expert system attempts to work
with information from near the periphery of the problem domain, it may yield
unpredictable results.  The problem domain must be readily defined using
empirical knowledge such as facts, rules, or algorithms.  It should not
require common sense (i.e., knowledge everyone takes for granted, but few can
articulate) or sensory data (i.e., vision or sense of smell).

          3.3.6  An Expert  Can Solve the Problem in Less than a Week

          The project manager or software developer should  also consider the
complexity of the tasks that are to be automated with an expert system.  The
tasks should neither.be too difficult nor too trivial for a human expert.  A
task requiring more than a week to solve without computer support is almost
certainly too large to be thoroughly delineated using rules.  However, if the
problem can be parsed into subproblems, it may be manageable.  On the other
hand, a task requiring only a few moments to solve might be automated more
efficiently using conventional technologies.

          3.3.7  A Significant Return on Investment is Anticipated

          Since expert system development requires a significant investment in
terms of people and money,  the expected return on  that investment must be well
understood, along with the means of measuring the  return.  Can the relative
 •

                                      3-4

-------
                                                     OSUER Directive «9028.00a

success of the expert system's performance be assessed?  The Initiation Phase
should address this issue before moving to the Concept Phase.


3.4       PRODUCTS

          Many products are produced and/or updated in the  course  of the
system life cycle.  The Initiation Phase products are described below.

          3.4.1    Initiation Decision Paper

          The Initiation Decision Paper describes the information  management
problem and justifies undertaking the next phase of the life cycle.  Whether
an expert system  is being considered as the solution or not, the Initiation
Decision Paper should be written in technology - independent language and
should focus primarily on the scope and magnitude of the problem at hand.  It
may highlight the attributes of the problem that are suggestive of a possible
expert system solution (refer to Section 3.3, "Success Factors").   The key
parts of the Initiation Decision Paper that will affect expert system projects
are:

          o        Mission areas addressed:  boundaries and domain  for  the
                   problem must be  clearly established.

          o        Description of the problem:   conventional ways of
                   describing information  management problems such  as flow
                   charts  and data  flow diagrams may be only partially  useful
                   for an  expert system-type problem.  Different  decision
                   [representation]  techniques include decision trees,  rules,
                   or  just plain English explanations  may  be preferable.

          o        Overall project  approach:  methodologies and tools may be
                   different for expert systems.   These issues  are  discussed
                   as  cross-cutting considerations in  Chapter 9.

          Prototyping may be used as a tool to develop the  analytic basis for
the decision paper.

          3.4.2	Project Management Plan

          Developing the preliminary Project Management Plan, the  second
product associated with the Initiation Phase, is an  important step because it
results in the collection  of the following information:

          o    •    Preliminary life cycle  cost estimate,

          o   -    Detailed estimate for the Concept Phase,
                                      3-5

-------
                                                     OSUER Directive 09028.OOa

          o        Results of threshold analysis of appropriate levels of
                   review and approval,

          o        Methodologies and selection of tools  to be used in the
                   Concept Phase, including prototyping,

          o        Results of benefit/cost benefit analyses,

          o        A list of experts who will provide the knowledge base of
                   the expert system, and

          o        A list of users who will ultimately be responsible for the
                   success or failure of the system.

          Techniques  for  justifying  expert  systems are  in many ways alike  to
those used to justify other information management systems.  Some of the
techniques used to justify expert systems are described below.

          o        Benefit Analysis - This approach takes advantage of
                   tangible and quantifiable expert system benefits and
                   attempts to quantify the qualitative benefits.  Depending
                   upon  the measures of success of the expert system defined
                   early in the development cycle, there may be significant
                   economical advantages to building an expert system.
                   Managers need to be aware of the applicability of this
                   approach to justifying expert system projects and the
                   measurability of benefits such as "improved productivity"
                   and "improved accuracy" such as what mistakes have cost in
                   the past.  Also, how would the resources be applied if the
                   expert system is not deployed?

          o        Cost  Savings Analysis - A second approach to justifying an
                   expert systems involves the cost savings.  While expert
                   system development may seem expensive, the cost of
                   remaining with the current system over time may .prove to
                   be more expensive.

          Depending on  the magnitude and  type of problem being considered,
several expert system methodologies and tools may be used in later phases.
These will be identified in the Concept and Definition and Design Phases.
Expert system-specific issues include life cycle adjustments including
consolidation of phases  and stages and use of prototyping techniques, and
special considerations for testing,  validation, and maintenance of knowledge
bases.  Refer to the OSWER Project Management Plan Practice Paper for more
information.
                                      3-6

-------
                                                     OSUER Directive *9028.00a

3.5      ACTIVITIES

         Activities that result in the products listed in Section 3.4 must be
completed by the project management and development team.  Involving the users
and gaining management commitment are two additional activities that must be
completed.

         3.5.1	Involving the Users
          »
         The Initiation Phase  requires a great deal of participation of
individuals not specifically involved in developing the expert system.  If
domain experts are not the ultimate users of the system, the actual users must
also be represented in describing the problem.

          3.5.2	Gaining Management  Commitment

         One of the goals of the Initiation Phase is to decide whether to
commit resources to solving the perceived problem.  This will entail
generating the necessary management support.  Failure  to acquire management
support now cannot.be compensated for  in later phases.  Therefore, high level
management from all involved organizations should participate in system
initiation.
                                      3-7

-------
                                                     OSWER Directive #9028.OOa
                                   CHAPTER  4
                                 CONCEPT  PHASE
4.0  INTRODUCTION

          The Concept Phase is the second of five major phases in the OSWER
system life cycle.  This phase of the expert system life cycle provides a
high-level description of the solution to the problem and describes the
functional, knowledge, and data requirements of the task and evaluates
alternative solutions.  A prototyping exercise may be used to develop the
concept, but it is not an end in itself.  The Concept Decision Paper is
separate from any prototyping activity.

          All aspects of the problem solution are considered in the Concept
Phase, and many can be addressed by attempting to develop a prototype.
Following are the questions which are raised at this time.

          o.        Is an expert system  the  best  solution  for  the problem?

          o      * What  type  of feasibility assessment is necessary?

          o        What  knowledge  is necessary to solve the problem,  and
                   where and  how  it can be  obtained?

          o        How should the  knowledge be managed?

          o        What  features  and functionality are expected of the  expert
                   system?

          o        What  combination of  knowledge representation and control
                   structure  best  suits the problem?

          6        How should the  project team be assembled?

          o        What  qualifications  must the  system developers and
                   reviewers  meet?

          o        How can the use of expert systems  to solve the problem be
                   justified?

          Because several key decisions are made in this phase, it is
important to review  the  objectives of the Concept Phase that  are described
below.
                                      4-1

-------
                                                                OSWER DIRECTIVE #9028.00e
                        rihAfcnano*
                         Deafen
Objectives
Decisions
 Determine the feasibility of an  expert  system solution
 to the problem

 Identify •  cost-effective system  solution to  the  problem
Approach  decisions  Include:
 - Is  the  problem important and  are expert systems  a viable
   solution
 - what Is  the system  life cycle  strategy
 -what are the  methodologies beat suited  to the project
 - what Is  the procurement plan
 - when and from whom will funding be  obtained

Execution decisions  Include:
 - what are the  high  level  functional and data requirements
 - what are  the  knowledge management  Issues
 - what Is  the knowledge  representation and control structure
  of   choice
 - who  la to be  on  the development team and what Is their role
 - who  are.  the  uaera  .
 - can  the  problem  be  aolved with existing  systems
 - what  is  the  overall  architecture
 • will  the  system  Interface  with  existing systems
 - what will  be  the  delivery  environment
 - how  will  technical,  programmatic, and other risks be
   addreased
 • what will  be  the  maintenance  strategy

Continuation  deciaiona  Include:
 • does the  Information management problem  continue to  exist
 - does the  concept  allow the development team  to  proceed
  to the  next  phase
 - are  sufficient  funding  and  other resources available
 [slew       *   System  Concept
 Products  *   Concept Decision Paper
            •   System  Test  Document
                                 Acceptance  Teat  Document
                                 Knowledge  Management Plan
                                 Data Management  Plan
                                 Requirements  Definition
                                 Figure  4.1
                                Concept  Phase
                     Objectives, Decisions,  and  Products

-------
                                                     OSUER Directive #9028.OOa

4.1       OBJECTIVES

          The objectives for the Concept Phase (Fig. 4.1) include a problem
definition, requirements, feasibility, and approach.  All of the objectives
evaluate or describe an approach to solve the information processing problem.
The first objective of the Concept Phase is to confirm the existence of the
information processing or knowledge-intensive problem.

          The second objective is to identify high level requirements for a
solution to the problem.  These requirements should focus on the nature of the
problem and the user's needs, and not directly address expert system issues.

          The third objective is to determine the feasibility of an expert
system solution to the problem.  This requires a study of both the
applicability of expert systems to the project and the capabilities of other
information technologies.  The final objective of the Concept Phase is to
identify a feasible, cost-effective expert system approach.  In order to meet
these objectives, several decisions must be'made concerning the approach to,
execution of, and continuation from the Concept Phase.
4'. 2       DECISIONS

          Decisions within the Concept Phase (Fig. 4.1)  focus on selecting
development methodologies, assembling resources,  and evaluating the expert
system approach to solving the problem.

          4.2.1 Approach

          There are five decisions to be made in the approach to the Concept
Phase.  The approach decisions select development methodologies and determine
the procurement plan.

          4.2.1.1 Approach Evaluation

          First,  is an expert systems a viable solution?  This emphasizes the
          need to study alternative 'approaches.

          4.2.1.2 Life Cycle Strategy

          Second,  what is the system life cycle strategy?  This decision
          includes the dependence of the project on rapid prototyping.

          4.2.1.3 Deve1opment Me thodolo gie s

          Third,  what knowledge acquisition, Development, testing, and
          maintenance methodologies are best-suited for this project?  It is
          important to be aware of the effects of each methodology on the
          development process, and choose methodologies best suited to the
                                      4-2

-------
                                                    OSUER Directive #9028.OOa

         application.  To some degree those choices may emerge from the
         prototyping approach.

         4.2.1.4 Procurement Plan

         The fourth decision is determining the procurement plan.  How are
         funds to be obtained and allocated for developing the expert  system?
         Does the project manager need to acquire any hardware or  software?

         4.2.1.5 Funding Determination

         Finally, what portion of the development cycle should be  funded at
         this point and by whom?  Is funding  available for the entire
         project?

         4.2.2 Execution

         There are several decisions to be made in  the execution portion of
the Concept Phase,  most of which focus  on the high level requirements of the
expert system and resource assembly.

         4.2.2.1 Functional'Requirements  Definition

         The  first decision during  execution  is determining  the  high-level
         functional requirements of the system. .This decision focuses on the
         nature of the problem  and  the users' needs rather than  on specific   •
         expert system issues.

         4.2.2.2 Knowledge Management Issues

         The  second decision  involves identifying the knowledge  management
         issues for  the project.  Are there  other expert  systems in the same
         problem solving  area that  might  provide  insight? How should the
         knowledge for this system  be acquired, placed  into  a knowledge base,
         and  maintained?

         4.2.2.3 Knowledge Representation and Control Structures

         Third, what  is the knowledge representation and control structure  of
         choice?  If  there are  no restrictions on the choice of  a  development
         environment  or expert  system shell,  which  of the knowledge
         representations  is best suited for  this particular  problem?   Given
         the  knowledge representation, which  control structure provides the
         best inferencing?  Refer to CHAPTER  5 for  explanations  of control
         structures.

         A.2.2.4 Data Requirements

         Fourth, what --if any --  are the data requirements and design
         parameters for this  system?  Is  the  expert system supposed to access

                                     4-3

-------
                                           OSUER Directive #9028. OOa

information from external data bases or programs?  If so, what is
the data's format, where does it reside, and how can it be accessed?
Some expert systems acquire all data by asking the user for
information.

4.2.2.5 PropfE Te>am Assembly
The fifth decision focuses on assembling the project team.  Who is
on the development team and what is their role?  It is important to
know both the qualifications and the availability of potential team
members .

4.2.2.6 User Needs Determination

The next decision involves identifying the users, their needs, and
their level of sophistication.  This is important for properly
conducting the feasibility assessment and determining the functional
requirements for the expert system.

4.2.2.7 Technology Selection

The seventh decision looks at the need for an information processing
system in general.  Can the problem be solved with existing systems?
If so, then it may be more effective to modify 'existing systems than
to build a new one. If an expert system is chosen, then which shell
or .language is appropriate?

4.2.2.8 Architecture Planning

The eighth decision entails planning the overall architecture of the
expert system, including the structure of the knowledge and the
data.

4.2.2.9 Integration Issues

To what degree the expert system will interface with or be
integrated into existing information-processing systems.
                           •
4.2.2.10 Delivery Environment Determination

What elements of the system will be centralized and what elements
will be distributed?  This is important in selecting a development
environment, determining licensing fees, and so on. •

4.2.2.11 Organizational Issues

How will technical , programmatic, and other risks be addressed?
Determining the procedures for handling risks also affects the
development process and the focus of the expert system.
                            4-4

-------
                                                     OSUER Directive #9028.OOa

         4.2.2.12 Maintenance Strategy

         The  final  execution decision is selecting  the maintenance strategy.
         Who  will maintain  the system?  How will users' comments be
         incorporated?  How often will  the  system be updated?  . Answers
         arrived at here  may be  changed later. Refer to CHAPTER 7 for  more
         details.

         4.2.3  Continuation

         There  are  three  decisions  in the Concept Phase that are a
continuation of decisions made during the Initiation Phase.  The continuation
decisions require an objective evaluation of the nature of the problem, the
proposed solution,  and the resources available to the project.

         4.2.3.1 Problem  Continuation

         First, does  the  information/knowledge management problem continue to
         exist? If yes,  proceed with the project as it  is currently  defined.
         If not, consider refocusing or discontinuing the project.

         4.2.3.2 Solution Adequacy

         Second, does the expert system concept  address  the problem
         sufficiently to  permit  continuing  to the Design and Definition
         Phase?  If it does,  then continue  to the next phase.   If  it- does
         not, then look for alternative expert system solutions --  including
         hybrid expert system/conventional  system applications --  or  consider
         other information processing  technologies.  Prototyping may  be used
          to seek answers  to these  questions.

         4.2.3.3 Project  Funding

          Finally,  are sufficient funding and other  resources available for
          the system life  cycle?   If resources are  available, proceed to the
         next phase.   If not,  look for  additional  resources or postpone the
         project until they become available.

         These decisions  are documented in several  new products.   The
         products  of the  Concept Phase  are  listed below.

*
4.3       PRODUCTS

         There are seven products  in the Concept Phase (Fig. 4.1):

          o         System Concept
          o         Concept  Decision Paper
          o         System Test Document
          o         Acceptance Test  Document

                                     4-5

-------
                                                     OSUER Directive «9028.00a

          o        Knowledge Management Plan
          o        Data Management Plan
          o        Requirements Definition.

          A.3.1  System Concept

          The System Concept is the key document of the Concept Phase.   It
describes the results of the functional analyses and both the data and
knowledge requirements for the expert system.

          4.3.2  Concept Decision Paper

          The Concept Decision Paper  should explain the benefits of selecting
the expert system approach that were determined in the Concept Phase.  This
document should be clear and comprehensive so that OSWER managers can make an
informed decision whether to approve a project.

          4.3.3  System Test Document

          The System Test Document presents information on  the testing  to be
performed by the development team.  It provides a chronology of the system
testing process, including strategy,  plan, data and knowledge, methods,
procedures, results, and recommended actions.

          4.3.4  Acceptance Test Document

          The Acceptance Test Document presents information on testing  to be
performed by OSWER program personnel.  At the end of the Concept Phase, the
Acceptance Test Document contains only the testing strategy.

          4.3.5  Knowledge Management Plan

          The Knowledge Management Plan describes the approach to acquiring,
utilizing, maintaining, and reusing knowledge throughout the project.  In the
Definition and Design Phase it will include the development team's choice of
knowledge acquisition methodologies,  knowledge representation, control
structure, and maintenance strategy.
                             *
          4.3.6  Data Management Plan

          The Data Management Plan reflects the project's data management
approach.  This document supplements the Knowledge Management Plan by
describing access to external data in databases, application programs, and
historical files.
                                      4-6

-------
                                                     OSUER Directive #9028.OOa

          4.3.7   Requirements  Definition

          Information from the experts  and users are compiled to form the
guidelines used in the Definition and Design Phase of the expert system
development life cycle.  Information to be included in -the Requirements
Definition specific to expert systems are:

          o        Target  level and focus  of output
          o        [Explanation facilities]
          o        User  interface
          o        External interfaces
          o        Target  performance  (speed and accuracy) of the  system.


4.4       SUCCESS FACTORS

          Several factors that affect an expert system's success should be
considered in the Concept  Phase.  They fall under the general topic  areas  of
organizational and resource issues, the target  users of the  system,  functional
requirements, and knowledge representation and  control structure.  Several
success factors  are  identified and general advice is given on applying them.

          The first major success factor is specifically identifying and
effectively implementing the  products of  the Concept Phase.   Proper
utilization of the Concept Phase-leads to  advantages such as assembling  an
efficient team,  good Requirements Definition and resource estimation, and
clear management direction.

          Another high-level success factor in the Concept Phase involves
implementing expert  systems around well-conceived ideas.  These can  be
developed through prototyping, but this is not  a substitute  for the  c'oncept.
Expert systems should be implemented to solve problems that  are not
effectively handled  using  conventional technologies.  This success factor  can
be ensured by carefully  studying the problem and looking for conventional
alternatives.

          Success factors relating to specific areas within  the Concept Phase
are described below.

          4.4.1 Organizational Issues

          There are several organizational issues that affect the  outcome  of
an expert system project.  Organizational  issues include scheduling
flexibility, management  commitment, available hardware and software
availability, and implications assessment.

          4.4.1.1 Scheduling Flexibility

          The first organizational issue is that the concept of an expert
          system is not critical to the solution.  This implies that there is

                                      4-7

-------
                                                     OSUER Directive #9028.OOa

          freedom for the project to change directions and  use a solution
          other than an expert system.   System flexibility  is important for
          expert system projects because they frequently need major
          modifications as ideas transform and new directions are discovered,
          either through prototyping or conventional analyses.

          4.4.1.2 Management Commitment

          Management commitment to an expert system project is often cited as
          one of the most common reasons for project success.   Management
          commitment comes in many forms:  resources -- including dollars,
          people, and equipment;  continued involvement and supervision;  and
          support in times of conflicting objectives.   Management needs to be
          aware of the benefits, limitations,  and differences between expert
          systems and conventional information processing tasks at the
          initiation of the project (see CHAPTER 3).   A constant flow of
          information and updates is required to keep management involved,
          interested,  and committed to the project.

          4.4.1.3 Impact Assessment

          Another organizational issue is that the ramifications of the expert
          system --on people,  the'task, or the organization itself -- are
          thought out in advance.  Take time to review the  implications of
          changing current processes caused by adding an expert system.

          4.4/2 Resource Issues

          Major resource issues stem from the proper evaluation and resource
estimation of expert system projects.  While this is a difficult task, proper
evaluation of the project goes a long way to ensure that the resources are
best allocated to an expert system, and would not generate higher payoffs on
other projects or other technologies.

          Resource estimation is a critical factor influencing the success-of
expert system projects.  It is  important that the developer accurately
estimate the time, staff, and financial resources required to complete the
project.  Another resource estimation factor is allowing for experimentation
or exploration through prototyping, both often necessary in the development of
an expert system.  In order to benefit  from these factors,  the project team
should evaluate  other expert system projects and set aside extra funds for
necessary exploratory work.

          4.4.3 Measures of Success

          Measures of success for the expert system project are clearly stated
and agreed upon. . Measures of success can be quantitative --  increased
productivity, or time savings --or they can be qualitative -- such as
improved morale.  These measures need to be identified and defined prior to


                                      4-8

-------
                                                     OSUER Directive #9028.OOa  '

the start of the project so that they are used to guide and evaluate the
expert system project.

          4.4.4  Target User Level of the System

          Success factors associated with the target user level of the system
involve an understanding of the users' needs and a specification of the
content and level of complexity of expert system's outputs.  Some effort
should be put  into determining how the expert system's recommendations will be
used.

          4.4.4.1 User Identification

          The first factor is identifying the intended users of the expert
          system.  This leads to a clear idea of both the focus of the output
          and its level of sophistication.  Users at an entry level position
          will require a different focus -- one that pertains directly to
          their task --as well as a specific degree of sophistication.  The
          output should be very specific and in terms that they can
          understand.  Advanced users, on the other hand, are often better
          served by succinct answers that they can use as guidance.

          Identifying the intended users is also essential when they vary in
          their degree of computer literacy.  Special help features may be
          necessary depending on the level of computer experience of the
          target users.

          Another success factor in determining the target output of the
          system is the degree of training that is necessary.  If the expert
          system is targeted entirely toward training, then it should focus on
          providing as much information to the user as possible.  Training
          systems often try to diagnose what problems a user is having with a
          concept, and then set up exercises to correct the problem.  Expert
          systems that are 'intended to have only incidental training benefits
1          focus on solving the problem with a minimum amount of overhead and
          their output tends to be more succinct.

          4.4.4.2 Content of System Outputs

          A second facet of this issue is understanding how the output of the
          system is to be used.  Once the users are identified and the target
          level of the output is set, how are the users to treat the expert
          system's recommendations?  Are the recommendations to play the role
          of a checklist, an assistant, or an expert?  It is critical that the
          users' of an expert system understand the degree to which they can
          rely upon the output of the expert system and the level of their own
          responsibility.
                                      4-9

-------
                                                     OSUER Directive #9028.OOa

          A.U.4.3 Level of System Outputs

          The final success factor in the target output involves specifying
          the underlying knowledge that will be incorporated into the system.
          Given the application,  what information is necessary to make an
          informed decision to accurately solve the problem?  It is important
          to gather this information from the expert before beginning the next
          phase.

          4.4.5  Functional Requirements

          It is important that functional requirements  are  carefully defined.
It is quite difficult to fulfill moving specifications under any circumstances
and especially given time and resource constraints.  The requirements should
be written, agreed upon, and modified only when the time and resources are
also changed commensurately.

          Justifications for the  expert system's recommendations need to be
clear and specific.  Explanation facilities -- such as the why and how queries
often found in expert systems -• often consist of replaying the logic used by
the system to arrive at a conclusion.  While this is sufficient for some
applications, others require more in-depth justification including causal
relationships and assumptions made.

          4.4.6  Knowledge Representation and Control Structure

          When an expert system shell is already selected before the Concept
Phase is completed, the choice of a knowledge representation and a control
structure is somewhat limited.  This poses a challenge to the developers when
the nature of the problem does not fit well with specific knowledge
representation scheme supported.  The feasibility study should show whether or
not alternate development environments would provide a cost-effective means to
successfully avoid this problem.

          The proper knowledge representation and control structures expose
the natural constraints of a problem and allow the expert system to be built
efficiently and easily.  At times, such as when the knowledge representation
is predetermined by the available expert system shell, there may be no
alternative but to proceed.  When there is a choice, the nature of the problem
should be used to select a suitable knowledge representation and control
structure.
4.5       ACTIVITIES

          There are several introductory activities to be undertaken during
the Concept Phase.  These involve three types of participants in an expert
system project - the users, the expert, and the project management.  These
activities will lay the groundwork for the major activities of this phase.
                                     4-10

-------
                                                          OSWER DIRECTIVE #9028.003
                      Knowledae  Representations
           RULE
IF the LIST has not been upgraded
 AND age is greater than 10 years
THEN upgrade must be performed
           FRAME
 [Storage
      AKO
 I
  Tank :
  AKO: Storage
  Has put: Liner
  Type: imdergroud
         IS-A
fank-1:
SA: tank
   ondition: good
  Type:
                Type:
                Material:  steel
                Contents: oil
                                               OBJECTS
                                 (Tank-1 XTank-2) (Tank-3) (Tank-4

                                     Messages: create, destroy,  upgrade
OAV Triplets
object
tank
tank
tank
attribute
type
type
...3Q9
value
steel
fiberglass
10
                                              Hybrid-Rule-OAV
IF NOT Tank:condition:upgrade
AND Age > 10 (parameter)
AND Tank:type:steel
THEN upgrade UST
                              Figure  4.2

-------
                                                     OSUER Directive #9028.OOa

          4.5.1  Identifying the Users

          Identifying the users of the application seems apparent,  but in fact
can be quite difficult.  [End user] identification is critical to proper
design and development because, several expert system features are determined
by the intended end users, including:                            .:

          o        The delivery  environment and geographic distribution,
          o        Target level  of the expert system's  recommendations,
          o        Type  and sophistication of the user  interface,
          o        Level of explanation in the justification mechanism, and
          o        The focus of  the knowledge base itself.

          The ultimate users will work with  the  expert system on a regular
basis.  Other people who will be indirectly involved with the use of the
expert system -- such as managers, support technicians, and maintenance staff
-- are not considered users.  A precise definition of the users leads to a
more focused and productive expert system.  .

          After all of the people who will be  directly involved in the
operation of the expert  system are identified, a profile of the group should
be developed.  This profile contains  information on the specific task that the
users need the expert system to perform, the level of job training and
experience of the users, and their range of computer literacy.  The conceptual
model of the expert system can then use this user profile to specify features
that focus on the specific needs of the intended users.

          4.5.2    Selecting the Expert

          Selection of the expert may seem like  an obvious  task,  but should be
given adequate consideration.  Several factors should be considered when
selecting an expert, or  determining if an appropriate expert exists.  This
generally leads to an expert who  is cooperative and easy to contact and
schedule appointments for  [knowledge  engineering] sessions.  The knowledge
engineer should also elicit support and commitment from the expert's superior.

          The expert needs to have the requisite qualities  to facilitate
knowledge engineering.   The expert should be methodical, consistent, and
articulate in dealing with the knowledge engineer.  With this type of expert,
it is easy to get the expert more interested in the technology.

          The expert must not be intimidated by  the thought of an automated
system doing their job.  Explain to the expert that the system is not meant to
be a replacement, but to aid the expert in making more  informed decisions or
to help people in the field where the expert might not  be available.  The
expert should understand what  is required.  It is the responsibility of the
knowledge  engineer to properly inform the expert what is expected.
                                     4-11

-------
                                                     OSUER Directive *9028.00a

          4.5.3  Communicating with Management

          Management support is crucial to the success  of  any  information-
processing project and especially for expert systems.  Many people are
uncertain of the capabilities and practicality of expert systems.  The way to
obtain and maintain management support is to provide good channels of
communication  in the Concept Phase and throughout the life cycle.

          In the Initiation Phase,  management needs to  be  informed of the
capabilities of expert systems: their benefits, limitations, and feasibility.
Expert systems should be presented to management in terms of improving
productivity in current work or in the capability to accomplish tasks that
were previously not feasible.  Introduction to expert systems is often best
accomplished via written material followed by a briefing.  This allows the
managers  to learn about expert systems at their own pace, with a minimum of
time commitment.  Next, management needs  to understand the specific expert
system application area.  This will improve the quality of decisions made
throughout the life -cycle of the expert system.

          Throughout the development process management needs  to be kept
informed  on the  status and  needs of the project.  It is best to use written
memos and reports to keep an audit  trail  of the development process  (see
Section 7°.6.9).  Another useful form of communication  is  the demonstration.
An expert system application already in use can be used to demonstrate the
general features of an expert  system.  Demonstrations  of working prototypes
should be given  for management throughout the development cycle to illustrate
the progress and the direction of  the  project.

          4.5.4	Management Commitment  and Understanding  of Prototyping

          Management commitment is the key to the success of an expert system
project.  There  must be a perceived value in  excess  of perceived cost and risk
in the project to retain management commitment.  Managers must  also be aware
of the different techniques used in developing expert  systems.  Techniques
used in expert system development  that may be unfamiliar  to managers include
knowledge acquisition, prototyping, and knowledge base verification.

          4.5.4.1 Initial Management Commitment

          The need for management commitment is obvious, but is often
          overlooked.  Initially,  management commitment involves approval and
          initiation of the expert system project.   Management commits the
          resources to begin the project,  including funding for hardware,
          software,  and human resources.
                                     4-12

-------
                                                     OSUER Directive #9028.OOe

          4.5.4.2 Continued Management Commitment

          Continued management support as development proceeds is sometimes
          more tenuous,  and in some ways more important.  Once the project has
          begun,  there is a tendency to reallocate people on the expert system
          project: when conflicts arise.   An important form  of management
          commitment is supporting the expert system project even when budget
          cuts or other resource constraints are imposed.   Once the project
          has started management needs to be objective when comparing the
          expert  system project to other information-processing projects, and
          not be  adversely influenced by the mystique of  expert systems.

          4.5.4.3 Prototyping Issues

          One of  the major benefits of prototyping techniques is their
          inherent suitability to expert systems.   Expert system prototypes
          can be  developed,  evaluated,  and modified relatively quickly and
          inexpensively.   This allows additional opportunities to be explored
          with minimum risk.   Managers need to be aware of  the ramifications
          of prototyping (see Section 1.7) and be prepared  to utilize
          prototypes to their fullest.

          4.5.4.4 Life Cycle Issues

          Because of the significant differences between  the development
          processes of traditional information processing projects and expert
          systems -- such as the disposable prototype,  management needs to be
          alerted to expert system life cycle issues.   Major issues include
          knowledge management, knowledge acquisition,  prototyping, and
          knowledge base validation.  Proper understanding  of these issues
          (see CHAPTER 1) will allow managers to reap the full benefits of
          expert  systems.


4.6       KNOWLEDGE MANAGEMENT

          "In the knowledge lies the power" is a quote by Randall Davis of the
Massachusetts Institute  of Technology, who has helped develop several  famous
expert systems such as MYCIN.  In this statement he emphasizes the  importance
of the knowledge placed-  in an  expert system.  Knowledge is more than
information, just as  information is more  than data.  Knowledge results from
the capability to use information.  A planned methodology is necessary to
properly gather,  use, and maintain knowledge for expert systems.  Knowledge
management comprises  six parts, described below.

          4.6.1  Knowledge Acquisition

          In the  knowledge acquisition process, we identify, locate, and
gather the information and mental processes used^o solve problems.  The first
step in knowledge acquisition  is to  determine what knowledge  is necessary  to

                                     4-13

-------
                                                     OSUER Directive #9028.OOa

solve Che problem, including any background information.  The required
knowledge acquisition information includes the:

          o        Knowledge necessary to solve the problem;

          o        Location of the knowledge -- including experts,  texts,
                   historical data, and test data;

          o        Selected knowledge acquisition methodology;  and

          o        Plans  for storing and manipulating  the knowledge.

          4.6.2  Knowledge Representation

          Knowledge representation is important for four reasons.   First, it
simplifies  the  task of accessing the acquired knowledge  in  the  system.
Second, a good  knowledge  representation scheme exposes  the  natural  constraints
of  the problem, which makes it easier to solve.  Third,  the knowledge
representation  is  the media in which the knowledge is  stored, updated,  and
modified.   Thus a  good knowledge representation will expedite maintenance.
Finally,  selecting the proper knowledge representation will facilitate
knowledge transfer by. making the knowledge transparent and  easy to  manipulate.
The information needed for the Knowledge Management Plan from this  part is  the
preliminary choice of knowledge representations for the expert  system.

          4.6.3  Knowledge Base Creation
                                                        •
          Knowledge base creation consists of taking the acquired knowledge,
placing it in a knowledge dictionary, transforming it  into  the  selected
knowledge representation, and placing the acquired knowledge into the
knowledge base.

          4.6.3.1 Knowledge Dictionary

          The knowledge dictionary for expert systems  is similar to the  data
          dictionary for databases.  The purpose for the knowledge  dictionary
          is two-fold.  First, it documents the structure of the knowledge
          base  contents.  This facilitates both testing and maintenance.
          Second,  the knowledge dictionary enforces consistency in naming and
          transforming knowledge.  This is especially  important when several
          programmers are working simultaneously on the knowledge base.

          4.6.3.2 Knowledge Transformation

          The transformation process is not always straight-forward because of
          the inherent restrictions and constraints of any  knowledge
          representation.  In addition, there are concessions that have  to be
          made  for operational purposes.  These concessions generally consist
          of additions to the knowledge base that are  solely for the purpose
          of controlling the flow of operation, calling external data sources,

                                     4-14

-------
                                                     OSWER Directive #9028.OOa

          or updating user interfaces.   These additions are necessary for the
          operation of the expert system,  but can cause confusion in later
          attempts to utilize the information in the knowledge base.
          Assumptions are also placed into the knowledge base based the way
       j;  the control structure will operate.  These assumptions add to the
       J  problems of maintaining or reusing the knowledge, and should be
          minimized.

          4.6.3.3 Knowledge Modularization

          It is particularly important to  properly modularize the knowledge
          base.  Knowledge about a problem domain can usually be decomposed
          into smaller components.   For example, in a diagnostic expert system
          for a piece of equipment, the problem might decompose into
          diagnosing the electrical system,  the mechanical  system,  and the
          structural portion of the equipment.  Each piece  of the problem can
          be placed in a separate module within the knowledge base.

          Modules within knowledge bases serve several purposes.   First,  they
          allow common information to be stored in one centralized area.
          Second, they allow the developer to think about the sub-problems
          independently and thus simplify  the development process.   Third,
          modules facilitate the verification process by isolating errors.
          Fourth, maintenance is simplified because updates and corrections
          are made to independent and easily identified parts of the knowledge
          base.  Finally, software reuse is promoted by separating knowledge.

          4.6.4  Knowledge Base Validation

          Validation of the knowledge base is necessary to  ensure proper
performance of the expert system.  A knowledge base validation plan helps
developers perform a comprehensive review of  the expert system and its
capabilities.  Information on when the validation is to occur, the type of
verification to be performed, and where the  test data is to come from should
be included in the plan.

          It is important to realize that  value can be derived from any expert
system project, even those that are not implemented.  The knowledge acquired
in building an expert system is at times more valuable than the expert system
itself.  This  is due to  the fact that the problem-solving process is now
documented, and can be reviewed and stored.

          4.6.5  Knowledge Base Maintenance

          The need for maintenance arises  from changing or  evolving user
requirements,  the need to correct  errors,  revisions  in regulations or
procedures, and  advances  in the state-of-the-art.  The maintenance plan must
recognize these  sources  of change  and be prepared to react accordingly.   Once
the changes are made, there should be a knowledge maintenance plan and


                                     4-15

-------
                                                     OSUER Directive 49028.OOa

configuration management plan for testing and validating the knowledge base to
ensure its continued usability.

          4.6.6  Knowledge Base Reusability

          The ability to reuse knowledge bases saves resources.  Knowledge
management should stress the reuse of knowledge to the extent that it is a
viable alternative.  A major portion of the creation of a comprehensive
knowledge base is the inclusion of background information and underpinning
knowledge.  Once this information is stored in the knowledge base, the
knowledge to solve the specific problem is easy to enter.  Because of the
relatively high ratio of background knowledge to application-specific
knowledge, there is an evolving process of reusing knowledge for expert
systems performing in the  same problem area.

          Knowledge reusability is still a research issue at this  time.
Several large  projects are underway to develop general-purpose knowledge
bases, but practical applications are limited in number.  There are  some
aspects of knowledge reusability that might be helpful to OSWER project
managers:

          o   .      Identifying other  expert  system projects  within the  same
                    problem-solving  area,

          o         Obtaining these  systems and their documentation,.

          o         Conducting an extensive litesature search,

          o         Building a-well-structured,  modular knowledge  base,

          o         Maintaining the  knowledge base,  and

          o         Using simple versus  compound knowledge representations.

          4.6.6.1 Other Potential Expert Systems

          The developer should be aware of other potential expert systems
          within the same  problem-solving area.  If others have been
          developed, try to obtain a copy of the knowledge base or the
          knowledge acquisition sessions.  'If other expert systems are
          planned, the knowledge -engineer needs to obtain as much general
          knowledge as possible in the sessions, document assumptions and
          paths not taken, and store the results of the knowledge acquisition
          in an easily accessible manner.

          4.6.6.2 Modular  Development

          The  second step  in practical knowledge reusability is to build a
          well-structured  knowledge base.  This implies making the knowledge
          base modular and as complete as possible.  Assumptions should be

                                      4-16

-------
                                                     OSWER Directive #9028.OOs

          recorded, and the entire knowledge base well documented both on-line
          and in hard copy.  Inclusion of procedural pieces and operational
          code should be kept to a minimum.   Finally, the knowledge base
          should strive for clarity rather than efficiency  or elegance.

          4.6.6.3 Maintenance Procedures

          The final step of knowledge reusability is in the maintenance of the
          knowledge base.  Maintenance should adhere to the same principles as
          the development, or its reusability will decrease over time.
4.7       FEASIBILITY ASSESSMENT

          A feasibility assessment is essential to understand the scope of an
expert system project and the costs and risks associated with it.  This
feasibility study parallels that of the LCt\ Guidance,  A feasibility
assessment should be conducted by the knowledge engineer(s) in conjunction
with the expert(s) and the intended users.  The issues involved in the
feasibility assessment process are technical in nature.  The objectives of
this process are to determine the suitability and to define the functional
requirements of the proposed expert system.

          4.7.1 Determining the Viability of an Expert System Solution

          The first step in determining the feasibility of an expert system
project is to determine the need for an expert system.  After the problem
itself is verified as being important, the approach to solving it is studied.
If other solutions exist, it is important to consider whether expert systems
would add sufficient benefits to justify their use.  Another consideration in
this step is the possibility of combining expert systems with other computing
techniques.  Problems involving forecasting, optimization, or voluminous
information management might be best solved using a hybrid approach.

         •4.7.2 Technical Issues

          Technical issues to consider in the feasibility assessment include
the nature of the problem, characteristics of knowledge management,
availability of expertise, software interfaces and integration, and
verification and validation.
4.8       KNOWLEDGE REPRESENTATION

          Knowledge representation is recognized as a crucial part of any
artificial  intelligence project.  Choosing  the correct knowledge
representation  facilitates  the development  and maintenance of an expert
system.  Figure 4.2 provides  a graphic representation of the following aspects
of knowledge  representation.


                                     4-17

-------
                                                     OSWER Directive «9028.00a

          4.8.1   Rules

          Rules  are  the most common form  of knowledge representation for
expert systems,  following a simple if-then format.  Rules are used to
represent the rules-of-thumb or [heuristic] used in solving problems.  People
find it easy to describe complex processes in simple steps using if-then
rules.  Rules are good for capturing procedural knowledge.  Because they
capture only one small piece of the problem per rule, rules are not applicable
to all problems.

          4.8.2   Frames

          Frames resemble database records in that they store chunks of
information together in fields.  They differ from records, however, by adding
class-subclass  structure, including more  information to individual fields, and
by incorporating procedural  inferencing mechanisms into the information.

          Frames are used to store knowledge that has an important structural
component (e.g., hierarchies of chemical  information.)  Frames  are often used
to store  case histories, because they store relevant information in the same
place which facilitates retrieval.

          4.8.3  Objects

          Objects are similar to frames,  except that they normally rely on
message-passing for communications  and are more  independent.  Objects  tend to
be used in expert systems in a limited form, as  scaled down versions of
frames.

          4.8.4  Object-Attribute-Value Triplets

          Object-attribute-value (OAV) triplets and parameters are a simple
form of knowledge representation used for storing rudimentary information.
They are  generally used  in  combination with other knowledge representations
such as rules.

          OAV triplets store information  in three parts.  The object portion
contains  the name of the primary concept that  is being represented by  the
triplet (e.g.,  an apple).   The attribute portion contains  information  on what
particular aspect of the object we  are referring to  (e.g., the  color).  The
value  stores the actual  information describing that  particular  attribute of
the object (e.g., red).

          4.8.5 Parameters

          Parameters are similar to the variables of various programming
languages in that they are  typed and contain values.  Different types  of
parameters include numbers,  strings, sets, lists, and records.  The  difference
is that parameters as a knowledge representation also contain information
about  their use that is utilized by the  expert system such as an explanation

                                     4-18

-------
                                                     OSWER Directive #9028.OOa

of what the parameter represents, questions to ask the user to find the value
for a parameter, and information on where the parameter is used.  Parameters
store basic knowledge that is neither procedural nor structured.

          4.8.6  Hybrids

          Hybrid knowledge representations are becoming more prevalent because
they make up for the deficiencies of individual knowledge representations.
Hybrid knowledge representations typically consist of a combination of rules
and frames, or rules and objects.  This combines a structural storage
representation frames or objects with action-oriented representation rules to
capture complex knowledge more easily across a broad range of problem domains.
4.9       CONTROL STRUCTURES

          More than one control structure may be applied to most knowledge
representations.  The  selection of a control structure depends on the. nature
of the problem, the knowledge representation chosen, and the desired
performance of the system.

          4.9.1  Forward Chaining

          [Forward chaining] applies  to the rule knowledge representation.  It
takes data and information, applies the appropriate rules from the knowledge
base, and presents recommendations.  Because of this, forward chaining is also
known as data-driven and event-driven-processing.  Forward chaining is best
for problems  that have a specific set of inputs and a large number of possible
outcomes including design and configuration applications.

          4.9.2  Backward Chaining

          Backward chaining also applies to rules, and works backward from the
conclusions, until it finds  a combination of rules and data that support a
given solution.  Backward chaining works best on problems that have a limited
number of possible outcomes and several inputs including diagnostic and
classification applications.  Backward chaining results in intelligent
questioning,  because only information pertaining  to the current hypothesis is
requested.

          4.9.3  Inferenc ing

          Inferencing is a general term that refers to several other control
structures, including  pattern matching,  [backtracking], constraint
propagation,  and search techniques.  ("These terms are defined in the glossary
in APPENDIX B.)  Variations on these techniques are used in expert systems
that use knowledge representations other .than rules, such as frames and
objects.
                                     4-19

-------
                                                     OSWER Directive #9028.OOa

          4.9.4  Hybrids

          Hybrid control  structures  attempt to reap  the benefits and overcome
the limitations of one or more individual techniques.  Common hybrid control
techniques combine forward and backward chaining, or chaining and pattern
matching.  The hybrids are typically applied to large problems and
applications using hybrid knowledge representations.


4.10      THE SYSTEM CONCEPT

          The conceptual  model contains comprehensive information  concerning
the details necessary to properly design the expert system.  Information
contained in the conceptual model includes;

          o         Specific problem area and  solution type,
          o         Knowledge  representation,
          o         Control  structure,
          o         External data sources and  interfaces,
          o         User  interface,
          o         Target output level, and
          o         Justification mechanism.

          The conceptual model should also be the basis for making a
preliminary- assessment of the expert system shell or development environment
subject  to  the  Definition and Design Phase  in Chapter 5.
                               •
4.11      PROOF-OF-CONCEPT PROTOTYPE

          Prototyping is an iterative process of design and redesign during  *
development.  Each  prototype  builds on the  successful completion of the
previous prototype.  Prototyping involves  several stages,  the first of which
begins in the Concept Phase.  The Proof-of-Concept  Prototype is initiated
during this phase.

          The Proof-of-Concept Prototype is a very small working model of the
expert system developed  to assess preliminary feasibility  of the problem
domain.  It  is developed  by following a narrow line  of reasoning for a specific
topic to its conclusion.  If  using  rules as a knowledge representation, the
number of rules should range  from 5 to 25.  This prototype  shows inherent
strengths and weaknesses of further development.  It will  answer the question
of whether  or not another technology or approach is feasible.   It also allows
the knowledge engineer to assess the framework, scope, interfaces and issues
borne out during knowledge acquisition and the design process.

4.12      ASSIGNING A. PROJECT TEAM

          The project team consists of everyone directly involved in the
development of  the  expert system.   The aspects of project  staffing  addressed
here are the size and composition of the project team.  The size and

                                     4-20

-------
                                                     OSWER Directive #9028.OOa

composition of the project team will be directly related to the nature of the
problem and the conceptual model developed earlier in this phase.  The number
of people in project management will remain approximately the same for expert
system development.

          4.12.1  Expert Involvement

          Experts can vary in number from one  to 10  or 50.   Fewer experts  are
needed on small projects that have adequate development time.  In the case of
specialized projects, only a few experts may be available.  Multiple experts
are used on large projects, and particularly those combining expertise in
several related areas.  Multiple experts might also be used in parallel on
projects that have limited development time.  It is important to not use all
of the available experts in developing an expert system.  Some should remain
external to the project until it is time to verify the expert system's
knowledge base.

          4.12.2  Knowledge Engineer Involvement

          Knowledge engineers are  key members  of the expert  system development
team.  There should be one senior knowledge engineer and one or more junior
knowledge engineers depending on the chosen knowledge acquisition methodology
and on the number of experts.  Some knowledge acquisition methodologies such
as the two-on-one interviewing technique require multiple knowledge engineers.
Except for extremely small expert system projects, there should be multiple
knowledge engineers to increase accuracy and completeness of the knowledge
acquisition and transformation process.  In projects where multiple experts
are involved at the same time, a guideline is to have two knowledge engineers
per expert.

4.12.3 Proerammer Involvement

          The number of programmers on the development team  depends on the
role of the knowledge engineer and the amount of conventional programming that
needs to be performed.  If the knowledge engineer Is capable of and available
for programming the expert system, then programmers per se may not be
necessary.  Programmers are important for integrating.the expert system into
existing operations and for adding external functions to the expert system
especially hybrid expert/conventional systems.  Estimating the number of
programmers is otherwise the same as with conventional systems.

4.12.4 Reviewer Involvement

          The number of reviewers  is generally higher for expert systems
projects due to the need for external experts for knowledge base verification.
                                     4-21

-------
                                                     OSUER Directive #9028.OOa
                                  CHAPTER  5
                         DEFINITION AND DESIGN PHASE
5.0       INTRODUCTION

          After  an appropriate problem domain has  been selected,  the
Definition and Design Phase begins.  This phase is critical because it lays
the framework for the development activity to follow.
5.1       OBJECTIVES

          The following objectives  of the Definition and Design Phase (Fig.
5.1) must be met in order to assure a smooth Development'Stage.

          o         Problem definition -  the definition  of the problem  domain
                    has  to be refined and reaffirmed.

          o         Development environment selection -  the  success of  the
                    Development Stage hinges on careful  selection of  the
                    development environment.  The goal is to select an
                    environment that  is a good match  for the problem  domain.
                    Refer to Section  5.5  for more details.

          o         Delivery environment  selection  -  the consideration  of the
                    delivery environment  is also an important objective of
                    this phase.  The  requirements of  the end users should be
                    incorporated in the early stages  of  system design.

          o         Evaluation of expert  system shells applicable to  the
                    problem domain.

          o         System design - ultimately, an  overall design of  the
                    system should be  achieved.

                           «
5.2       DECISIONS

          Several decisions  must be made  in the Definition and Design Phase
(Fig. 5.1).  The main decision involves  the choice of an appropriate
development  environment.  The importance  of a good selection at this stage
cannot be  stressed  enough.  Refer  to Section 5.5 for guidelines in the
"election  of a proper development environment.  Also, the targeted delivery
environment  must be carefully selected to provide a  smooth  transition  from
development  to implementation.  In addition, teams should be assembled to
design and develop  the  system.
                                      5-1

-------
                                                             OSWER DIRECTIVE #902B.OOa
                      BtflnKlon
                          lS»»lgn
Objectives
Decisions
 New
 Products
     Refine  the  problem  definition

     Select  a  development environment that Is compatible with
     the  problem domain

     Select  the  delivery  environment

     Achieve a  satisfactory  overall system design
Decisions  to  be addressed Include:
• selection of  an  appropriate  development  environment
• selection of  a  targeted  delivery  environment
- Identification  and  assembly  of  design team  members
• Identification  and  assembly  of  development  taam  members
    •   One-page Design of the System

    •   Detailed  Design of the System

    •   Development  Plan

    .  Test and  Validation  Plan

    •  Support  Plan

    •   Development  Environment  Evaluation
                                 Figure  5.1

                        Definition and Design Phase
                     Objectives, Decisions,  and Products

-------
                                                     OSUER Directive #9028.OOa

5.3       PRODUCTS

          There  are  six products  developed during the Definition and Design
Phase.  These include design documents, management plans,  and evaluation.

          5.3.1  One  Page  Design Document

          The One-Page Design Document is  a brief description of the expert
system.  It gives the expert systems scope, purpose, and a preliminary
assessment of the knowledge representation and control strategy which
applicable to the problem domain.  This document is described fully in 5.7.1.

          5.3.2  Detailed  Design Document

          The Detailed Design Document is  an expansion of  the One  Page Design
Document.  Further refinement is made of the items in the one page design
document.  Refer to Section 5.7.2 for a further discussion of this document.

          5.3.3  Development Plan

          This plan is derived from the detailed design document.  It outlines
the resources needed  for the Development Stage.  Also, schedules are firmed up
in this plan.

          5.3.4 Test and  Validation Plan

          This plan is developed  to specify how testing and validation should
be handled.   It will  contain a methodology  for designing tests  for  the expert
systems.  It will also list resources available during testing  and  specify
when  testing  and validation will take place.  Key people in  the validation
process are named in  the plan.

          5.3.5 Support Plan

     ;     This plan contains information on what support is available. All
resources and the time they are needed are  specified in this plan.  Refer  to
Section 2.5 for information on what resources are necessary  in  the  Definition
and Design Stage.

          5.3.6 Development Environment Evaluation

          This document outlines  potential development environments. It lists
strengths and weaknesses of each.  The factors that need to be  addressed in
this  document are primarily from the detailed design document.  Refer to
Section 5.5 for procedures for selecting a  development environment.
                                      5-2

-------
                                                     OSUER Directive #9028.OOa

5.4       SUCCESS  FACTORS

          During the Definition and Design Phase,  both the  development and
delivery environments must be considered as integral components of the system
design.  Some problems fit neatly into commercial shells, while others do not.
The various development environments offer a variety of mechanisms for
knowledge representation, control structure, and conflict resolution.  These
must all be evaluated with care to avoid problems later during the Development
Stage.  Common expert system design success factors are listed below.

          5.4.1  Validate the  Development Environment

           The desired development environment features selected in Section
4.11 are validated during this phase.  Various software, hardware, and
technical issues should be addressed in the selection of a specific product.

          5.4.1.1  Hardware and Software Availability

          The  first consideration in selecting a  development  environment  is
          determining the hardware and software currently available to the
          user base.   This impacts,  and may highly constrain,  the choice  of
          both development and delivery environments.

          5.4.1.2    Software Issues

          Explore  the wide variety shells first to try to find a match with
          the  problem domain.   Shells can provide  a quick solution to many
          problems.   Refer to  Appendix A for expert system  shell evaluation
          factors.   Do not attempt to use complex languages without proper
          training,  appreciation for the difficulty of the  task,  sufficient
          time for development,  or the availability of experienced
          programmers.

          If a shell has  been  selected as a development environment,  be sure
          that the tool is affordable,  easy to install, and easy to use.   An
          overly expensive tool dilutes the overall benefits  derived.  Choose
          a tool that can accommodate the problem.  Furthermore,  the tool must
          be able  to handle the necessary data.  A superior shell can be
          modified to provide  additional features.

          5.4.1.3   Hardware and Technical Issues

          Special  development  hardware may be required to run .the tool.  The
          additional cost must be considered in the selection process. The
          development environment should be able  to scale up  well.
          Integration with desired data sources,  programs,  and output
          interfaces should be readily accommodated.  Establish reasonable
          goals  for satisfactory performance in terms  of speed and memory use.
          Migration to a suitable delivery vehicle should be  easy and
          inexpensive.

                                      5-3

-------
                                                     OSUER Directive #9028.OOa
          5.4.2  Verification  and Validation

          Verification and' validation methodologies must be  determined at this
stage.  Verification techniques are the methods used to determine that the
expert system has been built correctly.  Verification techniques for the
software, knowledge base, and interfaces should be derived from the expert  -
system's functional requirements.  Steps taken and methods used to verify the
expert system need to be mapped from the components up through interfaces and
software module interactions.   The operational points of the expert system
that need to be validated -- such as scope, effectiveness, and compatibility  -
- are identified at this point.

          Validation techniques are the methods used to determine that the
expert system conforms to the functional requirements and can be used as
intended.  The validation techniques should also be identified in the
Definition and Design Phase.  Issues such as the need for external experts,
and types and location of test cases should* be thought out.

          5.4.3  Delivery Environment

          When establishing a design for the delivery environment,  there are
several things to be aware of.  These  include:

          .o         The environment's flexibility, availability,  and
                    consistency with equipment  currently being used,

          o         Licensing  fees- required for distribution of  the system.
                    Be  sure  ask the vendor  about licensing fees when  selecting
                    a  shell.

          o         Graphics requirements for  the Production System.

          o         Problems caused by excessive data'migration  from  the
                    mainframe  to  a PC.   If  it  appears  that there  will be
                    excessive  data migration,  perhaps  it  is  a case when a
                    mainframe  implementation should be used.

          o         The migration from the  development environment  to the
                    targeted delivery environment may  be  technically
                    impossible.   Careful selection of  development and delivery
                    environment will avoid  this problem.

          5.4.4  Rationale and Justification Features

          Several decisions should be made during this phase regarding the
necessary rationale and  justification features of the  development environment.
The s e include:
                                      5-4

-------
                                                     OSUER Directive #9028.OOa

          o         Specifying the extent to which help  is available  in the
                   software and in which areas,

          o         Clarifying what constitutes an effective rationale  for
                   posing questions and for issuing recommendations, and

          o         Determining a consistent method for  ranking
                   recommendations.

          5.4.5   User Interface  Issues

          The  end user interface  is  the most  important portion of the expert
system.  User involvement in selecting the desired features should be .included
as early as possible to ensure the success of the system.  Some issues to
address include:

          o         Incorporating interesting, user-friendly screens  and
                  - system features,

          o         Providing convenient data input for  the end user  without
                   the advanced editing functions needed  only in development,

          o         Ensuring adequate  system response time-, especially  for
                   calculation-intensive applications,  and

          o         Creating query interface capabilities  that support
                   flexibility and sophistication.


5.5       SELECTING A DEVELOPMENT ENVIRONMENT
                                                              *
          In the Concept  Phase,  a preliminary assessment of the  development
environment was made. During the Definition and Design Phase,  the actual
selection of the development environment is made.  Several factors are
involved in selecting a development environment,, including the:

          o  .       Choice of delivery environment (hardware, software, and
                   degree of integration with existing  systems),

          o         Type of problem including whether or not it  is a  hybrid
                   expert system/conventional system problem,

          o         Suitable knowledge representations, and control structures,
                   and

          o         Necessary interfaces.
                                      5-5

-------
                                                     OSUER Directive 49028.OOa

          g, j?. 1   Features

          Some of the  features  of a development environment are  outlined to
give a better understanding of how to select one.

          5.5.1.1  Declarative  and Procedural  Elements

          Knowledge bases  can be composed of declarative and/or  procedural
          elements. A declaration is a statement of fact  or  an  assertion that
          some data item has a  given value.  An example of a  declarative
          statement is:   "The average annual temperature at weather station  X
          is 66  degrees."   In contrast,  facts  may also  be  derived by
          procedural methods.  The above fact  could have been represented by a
          procedure to compute  the average annual temperature from raw data.
          A knowledge  base consisting solely of declarative statements is very
          explicit.  Such a knowledge representation is readily  updated by
          users  with minimal training.  However, procedural additions  can
          greatly increase the  power and flexibility of the knowledge  base.
          Procedures handle abstract data and  can allow for changes in
          information  without altering the knowledge base.

          An effective tool incorporates both  declarative  and procedural
          knowledge representations.  Development environments that offer
          frames often support  both types of representations.  Frames  are
          groups of slots- that  can contain declarative  statements of facts
          and/or procedural attachments .to determine facts.   (See Section
          4.8.2. for more information on frames).

          5.5.1.2  Grouping and Modularity

          Grouping and modularity are two related techniques  that allow for
          faster development and greater extendibility of knowledge bases.  If
          ah expert system application is very complex, the problem domain can
          be separated into modular knowledge  bases and the resulting segments
          grouped into specialized areas.  In  this manner, each knowledge base
          or group of knowledge bases can focus on a narrow aspect of the
          overall problem.  To  expand the system, the developer can alter a
          particular subset of the problem domain, while maintaining the
          integrity of the other knowledge bases.

          Validation of an expert system is a  much simpler task if each
          component, knowledge base is a manageable size and can be tested
          independently.  The chosen development environment  should include
          the ability to separate a large problem domain into smaller distinct
          knowledge bases.  Furthermore, it is necessary to have some means  of
          communication between the segments,  such as parameter passing or a
          [blackboarding]  mechanism.
                                      5-6

-------
                                           OSUER Directive #9028.OOa

5.5.1.3  Documentation Needs

Clear documentation of the knowledge base is extremely important
during the development of an expert system.  Some tools represent
the knowledge base in a very readable, English-like format that is
almost self documenting.  The developer mus: be able to review the
knowledge quickly and easily at all stages of development.  Also,
concise documentation is necessary for validation and maintenance,
especially if these tasks are to be performed by someone other than
the developer.

5.5.1.4   User Issues

Another factor involved in the selection of a development
environment is its ease of use.  This is the result of the user
interface, and is determined by the type of interface provided by
the development environment and the effort of the developers.

5.5.1.5   Developer Issues

There is often a trade-off between the sophistication or power of an
[expert system development environment] and its ease of use.  In
other words, expert system computing environments that are best-
suited for difficult problems are often more difficult for the
developer to use.  It is important to take into consideration the
complexity of the problem and the skill of the developers when
selecting a development environment. Trade-offs between development
and run-time environments should be made explicit.

5.5.1.6  Knowledge Management Issues

The developer needs to be able to describe the knowledge represented
in the system in order to interact with the expert.  A careful
consideration of what has been included in the knowledge base is
essential before any expansion takes place.  The current knowledge
should be outlined and discussed to expose weaknesses or gaps in
reasoning.  The knowledge gaps can then be addressed in future
development.  A tool that cleanly breaks out the knowledge into a
hierarchy or decision tree will be of great help in this regard.
Also, some tools incorporate data element dictionaries.

5.5.1.7  Rationale

Most development tools allow the user to ask the system for its
rationale for asking a particular question during a consultation.
The user need only enter "Why?" in response to a system prompt.  The
system will generally show its current line of reasoning and explain
why it needs the information in question.  The user may choose to
answer the original prompt or perhaps respond with "Unknown" if no
                            5-7

-------
                                                    OSWER Directive #9028.OOa

         information is available.  The ability of a system to allow a series
         of "Why?" inquires to trace the line of reasoning has great value.

         5.5.1.8  Justification

         A common feature of development tools is the ability to explain the
         justification for the recommendations of a consultation.  The user
         can ask "How?" after a consultation to get a detailed trace of the
         line of reasoning used to arrive at the given recommendation.  This
         trace can be in both text and graphic format, and often allows the
         user to change one or more responses to experiment with a "What if?"
         scenario.

         5.5.1.9  On-line Help

         Many expert system shells offer on-line help both during development
         and during a consultation.  For example, the user can ask the system
       ~ for syntactic information while developing the  knowledge base.  This
         feature is especially useful for the novice user who needs to learn
         a tool quickly.  On-line help can also guide an experienced
         knowledge engineer through the more complicated tools.  Furthermore,
         the end user may require on-line help in order  to properly respond
         to questions during  a consultation.

         5.5.1.10  Explanation Facilities

         A common feature in  many development tools involves  some type of
         conclusion explanation  facilities.  During a consultation, this
         feature is used to describe the line of reasoning that  the system
         followed in reaching a  conclusion.  The developer may also wish to
         use a  consultation tracing feature  to aid in debugging  the knowledge
         base.

         5.5.2  User Requirements

         When selecting a development environment, it is important  to take
into consideration what the user requirements are.   Remember that user
acceptance will ultimately make or break your application.

         5.5.2.1  Developer Requirements

         The level of  the developer's computer skills will be a  determining
         . factor in the selection of a tool.  Some tools  are  designed  for  the
         novice user while others offer complex  features that an advanced
         programmer will utilize.  For example,  a beginner's  tool  is  usually
         restricted to a specific control structure with a simple editor  to
         create the knowledge base.  However, an advanced user may  wish  to
         experiment with various inferencing techniques  and  knowledge
         representations.  Generally, the more flexible  tools have  steeper
         learning curves.  For this reason  it is recommended that a new

                                     5-8

-------
                                                     OSUER Directive «9028.00a

          developer start with a more structured shell to gain experience with
          expert system technology rather than becoming entangled in the
          details of a complicated development environment.

          5.5.2.2  End User Requirements    !;
          ™™^MBA_^^MM«M_^_l   	••^^^•*^_^^_V*«a_MM_    |,,

          The end user's  requirements should also be considered in tool
          selection.  How well does the user understand the  problem domain?
          How computer literate is the end user?  The answers to these
          questions will  determine what level of on-line help and reasoning
          explanation the final system should offer.  Furthermore,  the
          delivery system should be more [robust] if the target user is less
          experienced.  If incorrect or incomplete data is given during a
          consultation, the system may need to inform a novice of the
          inconsistencies.

          5.5.3  Migration to Delivery Environments

          The target delivery environment should be considered early in the
system design.  First, the available hardware in the user base needs to be
determined.  It may be necessary for the end users  to buy new hardware or to
upgrade their current systems.  A more cost effective solution could be to
choose a shell that is compatible with the user's system.  Many tools are
designed solely for the PC environment and do not allow portability to a
mainframe.  Other tools are set up to run only on mainframes or dedicated AI
environments.  If the expert system is to interact with an existing system,
special interfaces may be necessary.  Some shells offer a link-up capability
with databases and external files with data-passing functions.  The more-
flexible tools allow  the user to customize the shell with specialized routines
that integrate with other languages.

          5.5.4  User Interfaces

          Several different types of user interfaces must be taken into
consideration during  the Definition and Design Phase.-

          5.5.4.1  Developer Interface

          There are a wide variety of developer interface features available
          from different tools.  The ease-of-use of a development environment
          is a function of the type of features employed in  the system.  Below
          are some of the features that can be found in the  developer's
          interface:

          o         Specialized editors  that  guide  the  user  through  the
                    creation of a knowledge base,

          o         Decision tree tracing -   a graphic  or  textual  listing of
                    questions  and responses,  and
                                      5-9

-------
                                                     OSUER Directive *9028.00a

         o         Debugging capabilities with a split screen showing the
                   inferencing during a consultation.

         5.5.4.2   End User  Interface

         End user  interface features also vary  from  tool  to  tool.   The
         features  required  will  depend on the target user as described in
         Section 4.4.4.   Some  of the desirable  end user interface  features
         include:

         o         Help features  - the  "Why" feature can be used during a
                   consultation  to have the system explain why  the  current
                   question is being asked, and the  "How" feature can be used
                   after a  conclusion has been reached to show  the  line of
                   reasoning,

         o         Active images  - these can display values  and scenarios  or
                   allow the user interact p-ictorially with  the system during
                   a consultation, and

         o         Re-run consultations - these will save the responses in a
                   consultation  and allow the user to change a  set  of
                   parameters  to see the outcome in  a "What  if" situation.

         5.5.5 Vendor  Information

         Information supplied by vendors plays  a significant role  in the  •
design and eventual development of the expert system.  Given below  are several
areas which must be considered.

         5.5.5.1.  Stability

         An initial  consideration in tool  selection  is  the stability of the
         vendor.   A  new,  small company may have an interesting product,  but
         may not have as  an impressive a track record as  a more established
         firm. Keep in mind that all  post-purchase  customer support is
         inherently  dependent on the vendor's  longevity.   The following
         questions should be posed  to  assess the vendor's stability:

         o         How long has  the vendor been in business?

         o         Is the vendor in good financial standing?

         o         How many systems have been installed?

         o         What types  of organizations have  purchased the  tool?

         o         How have other organizations used the  tool?

         o         Are there user's groups for the tool?

                                     5-10

-------
                                            OSUER Directive #9028.OOa
o         Are other users satisfied with the product?

o         What is it about this tool that makes it worthwhile to
          invest in it? and.
                          ^
o         What are the risks of developing the application with this
          tool as compared to all others?

5.5.5.2  Training

Training issues are especially important in the choice of a vendor
and shell.  Expert system development tools range from easy-to-use
yet limited shells to complex, powerful environments.  If the latter
is best suited for the problem domain, proper training will be an
essential contribution to the success of the project.  The danger
lies in the flexibility of the more powerful tools.  Such
environments are geared toward research and allow the developer to
select from a variety of control structures.  Training on these
systems is necessary to prevent the user from implementing the wrong
knowledge representation or inferencing strategy.  Vendors offer
varying levels of training, with the most extensive generally
reserved for the complex and expensive tools.  The types of training
available include:

o         Training manuals - should be easy to comprehend and geared
          for the user's level of expertise

o         On-line tutorials -  can be very useful in exploring the
          basic features of the tool
                                                     t
o         Training courses - offer a deeper understanding of the
          system's implementation and can be tailored to the user's
          problem domain

o         Consultants - may be needed to assist the developer with
          complex applications yet are available only through the
          larger,  more established vendors.          '

5.5.5.3  Cost

Cost is also a deciding factor when choosing a vendor.  Often, in
addition to the price of the initial development tool, there are
other potential costs which should be considered.  Listed below are
common items associated with expert system development environments
whose costs must be considered.

o         Initial development system,
                            5-11

-------
                                                     OSWER Directive #9028.OOa

         o         Runtime copies for the end users, or royalties for use on
                   a network,

         o         Additional AI language interfaces that may be needed to
                   run the development package,

         o         Specialized AI hardware (e.g. LISP workstations) that is
                   necessary for some of the more powerful tools,

         o         Update/upgrade for new versions; often the initial cost
                   can be applied toward that of the upgraded system,

         o         User support provided by the vendor's trouble-shooting
                   hotline, and

         o         Training, as shown in Section 5.5.5.2 in order of
                   increasing expense.

         5.5.6  Documentation

         The documentation of the development environment  should be both
comprehensive and easy to understand.  The more powerful development tools
generally have extensive documentation.  Many tools offer tutorials that guide
the developer through example prototypes.  A few environments have on-line
documentation.                                                    «

         5.5.7  Overall Evaluation

         An overall evaluation of the development environment should include
all of the items discussed above.  Special consideration should be given to
the cost of  the tool and associated hardware versus the tool's functionality.
In comparing various environments, try to rank the features in order of
importance for the given application.  Be sure to have a well defined problem
domain, formalized during the Concept Phase, before attempting to select a
development  environment.                                              ;


5.6      DEVELOPER QUALIFICATIONS

         In Section 4.12,  the task of selecting a project  team was initiated.
The aspects  of project staffing that were emphasized were the size and
composition  of the project team.  This Section will focus on the technical
qualifications of the team members.

         5.6.1   Knowledge  Engineer                  ,

         The knowledge  engineer is responsible for helping to design,  build,
and validate the expert system.  As such, the knowledge engineer should have
the following qualifications:


                                     5-12

-------
                                                     OSUER Directive 19028.OOa

          o        Good communications skills, particularly in listening,
                   interviewing, and writing,

          o        Basic computer skills, knowledge of requirements analysis
                   and design techniques,

          o        Fundamental knowledge of expert systems and artificial
                   intelligence in areas such as knowledge representation and
                   user interfaces, and

          o        Some expertise in the problem domain.

          Desirable qualities also  include  the  ability to work well with
people, experience in expert system development, a minimal background in
psychology, and experience in systems analysis.

          5.6.2   Programmer

          The programmer  should  have  fundamental programming  skills and
experience in the selected expert system development environment.  Desirable
qualities include familiarity with artificial intelligence techniques, system
integration, testing, and validation.

          5.6.3   Reviewer/Approver

          The reviewer/approver  should  be familiar with expert systems
technology and LCM issues, cost justification techniques, and other forms-of
information-processing projects.

          5.6.4   Expert

          The expert should be a recognized leader  in the field  in which the
expert system is being applied.   Desirable qualifications include an interest
in the project, availability, and good communications skills.


5.7       DESIGN
                                                              /
          The design is  important to  the  success of the expert system
application as it will be a guide for the Development Stage and the additional
prototyping which takes place there.   There are two steps to  the design
process.  Each has a specific purpose.

          5.7.1  One-page Design

          The One-Page Design Document  is a very high level view of the
system.  It is completed at the beginning of the Definition and Design Phase
after preliminary knowledge acquisition has taken place.  Its purpose is to
give a starting point to the design by giving brief description of each of  the
components of the expert system.  It should be emphasized that this document

                                     5-13

-------
                                                     OSUER Directive *9028.00a

should be limited to one page, hence the name.   This document is broken down
into the following six components :

          o         Overview of the system:  This paragraph briefly states the
                   purpose and scope of the system. It also gives an estimate
                   of the measures of success of the system.

          o         Components and structure:  A list of all components or
                   subdomains of knowledge and a brief description of how
                   each fits  together in the overall structure.

          o         Interfaces:  A description of any known interfaces
                   including  any database and external interfaces.  It also
                   states how the user interface will be established.

          o         External calls and data sources;  An enumeration of all
                   external calls the expert system has to make and any  data
                   sources which must be utilized by the expert system.

          o         Knowledge  representation:  A best approximation as  to the
                   appropriate representation, based on what  is currently
                   known about the problem domain.  It also includes reasons
                   for  the selection, because they  may be useful  later in
                   this phase and on into the Development Stage.

          o         Inferencing mechanism:  The type of inferencing mechanism
                   is determined primarily from the knowledge representation
                   scheme.  Refer to Section 4.9 to learn more about how to
                   select an  inferencing mechanism.

          5.7.2  Detailed Design

          The Detailed Design Document further elaborates what was borne out
in the One-Page Design Document. In this document,  the system is broken  out by
expert system component or context.  Within each component these questions
should be answered:

          o         What is the purpose of the component?

          o         What is the scope of the component?

          o         What information is the component going to need?

          o         Are  there  any specific external  calls and/or data  sources
                   needed?
                                     5-14

-------
                                                     OSUER Directive #9028.OOa

5.8       PROTOTYPING STEPS                                        x

          The prototyping which began in the Concept Phase continues during
this phase with the refinement of the Proof-of-Concept Prototype and the
development of the demonstration Prototype.
                   i '•
          5.8.1  Proof-of-Concept Prototype

          The proof-of-concept prototype is a small working system designed to
provide a preliminary feasibility assessment on the problem domain before
additional resources are allocated.  This prototype is started in the Concept
Phase, but is refined and achieves maturity here.  Ideally, this prototype is
designed to include only the top level features of the targeted system.  For
example, the goal recommendations of the knowledge base can be represented
without the full line of reasoning to handle all possible cases in the problem
domain.  The Detailed Design Document is merely alluded to and need not be
included at this stage.

          5.8.2  Demonstration Prototype

          The Demonstration Prototype is an extension of the Proof-of-Concept
Prototype.  The Demonstration Prototype should be small and specialized, based
on a narrow subset of the overall problem domain.  In contrast to the Proof-
of-Concept Prototype, an essential element is that the knowledge base is
designed to be deep in one area while maintaining breadth in the other areas.
In this prototype, more attention is paid to user interface features to make
the system attractive to management and to the end users.  Also, -the necessary
data requirements are addressed at this stage to enable the assessment and
design of the external data integration.
                                     5-15

-------
                                                     OSUER Directive *9028.00a
                                   CHAPTER 6
                              DEVELOPMENT STAGE
6.0       INTRODUCTION
          Tile Development Stage is a translation of the design into a working
system.  This chapter describes several success factors needed during
development, the types of prototypes produced, and knowledge acquisition.


6.1       OBJECTIVES

          There are several key objectives in the Development Stage (Fig.
6.1).  One is to document key  issues to be borne out during prototyping.
These issues are discussed for each prototype in Section 6.10.  In the
Development  Stage key issues for  the Implementation Stage should be
documented.   The problem description, functional description and knowledge
are refined  during development of the prototypes.  Now  is the time to increase
product visibility.  Also, maintaining management commitment is a key
objective of the Development Stage.  Another objective  is to codify  the
knowledge base and maintain its accuracy  through testing.


6.2       DECISIONS

          There are several key decisions that must be made during the
Development  Stage  (Fig.  6.1).  The first  decision is how-to proceed  given
current funding levels.  Before each prototype  is begun, two questions must be
resolved.  The first is, "To what degree  has knowledge  acquisition taken
place?"  The second question is,   "Is the prototype addressing the problem
sufficiently to be fully developed?"  User and  management commitment are very
important to the success of the project so decisions must be made  to satisfy
both user and managerial requirements.  Another decision that must be
addressed is whether the developer has identified and  resolved security  and
backup issues.      •    :
6.3       PRODUCTS                                           '  .

          There are several products associated with the Development Stage
(Fig. 6.1).

          o         Project management plan
          o         Test  and validation plan
          o         Support plan
          o         Knowledge  notebook
          o         Full  Prototype
          o         Production System
          o         Development documentation
          o         Development decision paper.

                                      6-1

-------
              System UfeCyde
                   for
               Expert Systems
                                                                         OSWEP DIRECTIVE #902B.OOa
                and
                                     Design
Irnpiornerttaton
            Objectives
•  Document key  Issues to be borne  out  during prototyping

•  Document key  decisions for the Implementation  phase

.  Perform  Implementation planning  before  full  Implementation

.  Refine  knowledge

•  Refine the  problem description  and functional description

.  Obtain  user acceptance

•  Increase  product  visibility

•  Maintain  management  commitment

.  Codify  knowledge

•  Maintain  knowledge accuracy  through testing
                         .  Declsons to be sddressed Include:
                            • how  should development  activities  proceed  given  current
                              funding   levels
             Decisions      - has the  prototype addressed  the problem  sufficiently to  be
                              Implemented
                            - to  what  degree  has  knowledge  aqulsltlon .taken place
                            «has the  project  gained user and management  commitment
                            ° have security and backup  Issues been  Identified  and resolved
            New
            Products
   Project Management  Plan
   Test  and  Validation  Plan
   Support  Plan
   Knowledge  Notebook
Full  Prototype(s)  (If  needed)
Production  System
Development  Documentation
Development Decision Paper
                                            Figure  6.1

                                         Development Stage
                                Objectives, Decisions,  and  Products

-------
                                                     OSUER Directive #9028.OOa
          6.3.1 Project Management Plan

          The Project Management Plan is  used to make  sure the Development
Stage is run smoothly.  It specifies who is in charge of each step of
development.  It also specifies the resources each person has control over and
outlines the hierarchy of management when there are conflicts in the decision
making process.

          6.3.2 Test and Validation Plan

          The Test and Validation Plan is developed to specify how testing and
validation  should be handled.   It will contain a methodology  for designing
tests for the expert systems, and it will list resources available during
testing as  well as  specifying when testing and validation will take place.
Key staff involved  in  the validation process are named at this time.

          6.3.3 Support Plan               '       .

          The Support Plan contains information on what support services are
available.  All resources and the time they are needed are specified in this
plan.  Refer  to Section  2.5  for information on what resources are necessary in
the Development Stage.

          6.3.4 Knowledge Notebook

          The Knowledge Notebook is used throughout the Development Stage for
knowledge acquisition  and is kept by  the primary knowledge engineer.  Refer to
Section 6.9 for a description of this  document.

          6.3.5 Full prototype

          The Full Prototype is an intermediate product in the Development
Stage.  This  prototype is described  in Section  6.10.1.

          6.3.6 Production System
                                                 i
          The Production System is the primary.product of this stage.  This
stage is not  complete  until  the Production System  is  produced.  The  rest  this
chapter outlines the  steps necessary  to  produce  it.

          6.3.7 Development Documentation

          The Development Documentation is produced while the prototypes are
being developed.  It  contains information on how each prototype was  developed
and documentation on  the knowledge representation  and control structures.  It
also contains the documentation for  any  supporting software of the expert
system.
                                      6-2

-------
                                                     OSUER Directive #9028.OOa

          6.3.8  Development Decision Paper

          The Development Decision Paper is  a documentation of all decisions
that were made during this stage.  It lists each decision and the reasons and
facts supporting why the decision was made.


6.4       SUCCESS FACTORS

          There  are several areas that are crucial  to the success of the
Development Stage.  These factors are grouped below according to the area they
address.    /

          6.4.1   Knowledge Engineering

          Knowledge engineering begins in the Concept Phase.   The knowledge
engineers should devise a method to resolve conflicts among multiple sources
of expertise with differing specialties.  Methods for dealing with conflicts
are given in Section 6.8.

          It is  wise to coordinate programmers and  knowledge engineers,  but
keep in mind that there may be organizational separation and differences in
background.  Knowledge engineers should have a strong computer background to
facilitate communication with programmers.  Knowledge engineering is most
effective when proven knowledge acquisition methods are used.  These include :

          o         Unstructured interviews,
          o   •      Structured  interviews,
          o         Observation,
          o         Interruption analysis,
          o         Constrained-processing tasks,
          o         Questionnaires,  and
          o         Decision trees  and decision tables.

   '       Many good techniques are available.  The  knowledge engineer should
apply one or several structured and unstructured techniques to document the
expert's  [domain knowledge].  These techniques are discussed in  detail in
Section 6.9.2.

          The knowledge engineers and expert should think out all of the
ramifications of the rules  on  a periodic  basis.  There should be development
of knowledge throughout  the project to  allow for constant refinement and
experimentation.   It is  a good idea to  periodically  recheck the -accumulated
knowledge in order  to better refine it.

          Knowledge engineers need not be experts in the field; too much
domain knowledge has a risk of producing  biases in the process toward the
knowledge engineers and  away from  users.  Select knowledge  engineers who are
familiar with the  domain,  but  are  not necessarily  expert in it.


                                      6-3

-------
                                                     OSUER Directive #9028.OOa

          Knowledge engineering  time  estimates  should be well  thought out with
respect to reaching the appropriate depth of knowledge.  There should be a
constant reminder to the project manager to plan for contingencies and expect
repeat sessions with experts and users.

          6.4.2  Prototyping Methodologies

          Below are several methods that contribute  to  a  successful prototype.

          o        Use rapid prototyping with  frequent interim  deliverables
                   and decision points.

          o        Discard the prototype if a  better design approach is
                   discovered.  The  purpose of a prototype is to  firm up the
                   design  specifications.

          o        The design should be modular to  show all lines of
                   reasoning and specific  functions  the expert  system is
                   required to perform.  The prototype should also be
                   attractive as a demonstration vehicle to gain  the support
                   of users and management.  The development team should
                   accurately define and adhere to  the system development
                    stages.   This will  allow the team to anticipate the long-
                    term  impact of  schedule deviations.

          6'.4. 3  Validation Process
                         •
          It is important to identify and validate external interfaces as
early as  possible.  This includes  inputs and outputs, and other programs, and
algorithms - separately  from the knowledge bases.  Validation should be  done
frequently and continually.  There is  a tendency to  ignore validation until
the end of the Development Stage.  Ideally, it should be done after the
addition  of each rule, but later can be reduced to  the end of each session.
Expert system  validity relies heavily  on the validity of accessed data.
Issues to be considered when validating include the knowledge base,
recommendations, justification, rationale, type of  inferencing  and incomplete
data.
                                                             I
          It is not always  possible to test all possible  rule outcomes.   This
is often  the case  for larger systems,  e.g, +500 rules.  This emphasizes  the
importance of  keeping a  "trace" of each session and  the need for  software
maintenance in the operational phase.

          6.4.4  Testing Procedures

          Testing is the most important part of the Development Stage because
there may be potential conflicts in  the knowledge base.  A thorough analysis
should be performed on a routine basis throughout development and upon
completion of  development.   Remember to test all possible rule  combinations to
identify  conflicting or  overlapping  rules.  Some ways to do this  are :

                                      6-4

-------
                                                     OSWER Directive #9028.OOa
          o        Random tests by beta users,                               •
          o        Selected test cases run  through the  system, and
          o        Ad-hoc testing by  the expert.

          Make sure to adequately test critical components and any examples or
rules generated by inductive systems.  Thoroughly retest entire knowledge base
when changes are made.  The process of testing expert system shells is easier
than testing implementations written  in LISP or PROLOG because the inference
engines have already been thoroughly  tested in a shell.  Remember not to
underestimate the proper  level of testing.  Allocate sufficient time and
resources to do the testing.


6.5       ACTIVITIES

          Two background activities  should be  completed during this  stage  to
ensure a smooth development of the expert system.  These activities include
the end users and management staff.

          6.5.1	Demonstrations and Briefings to Users

          Periodically, and throughout the Development Stage,  conduct
demonstrations for the users.  In addition,  the user should play a primary
role in the development of the user interface in order to assure acceptance of
the expert system in its  delivery environment.  User involvement here will
ensure ongoing support-for the expert system application.  Frequent
demonstrations of the system and milestone reports to management for the
expert system application are necessary for continued user involvement.

          6.5.2	On-going Progress  Reporting to Management

          Management  should be  updated on a  continual basis as to  the  progress
of the development effort.  If time or cost overruns are expected, management
should be given a detailed report.  Using detailed progress reports, it is
advisable to demonstrate  to management that as the project changes and the
requirements are refined,  the effort  Continues to be manageable and important.
                                                              y
                             •
          On-going management commitment is  vital to  the success of the expert
system.  To ensure management commitment, detailed progress reports should be
given weekly or in an appropriate time frame for the project.  Delays should
be attributed to discrete events and  not to the expert system technology
itself.  If proper feedback is given, management will be more confident that
the development effort is under control.
                                      6-5

-------
                                                     OSWER Directive #9028.OOa

6.6       KNOWLEDGE ACQUISITION

          Knowledge acquisition is the  process  of extracting and representing
the expert's knowledge in the form of a conceptual model.   A point which is
sometimes overlooked in acquiring knowledge is the fact that there are usually
multiple solution paths to reaching a conclusion, based on alternative
hierarchies of assumptions.  Expert systems must be designed to deal with
these types of uncertainties, which vary depending on the domain being
examined.  Experts are not always explicitly aware of precise concepts and
representations of their knowledge.  Documenting this knowledge requires
considerable patience and cooperation between the knowledge engineers and the
expert.  In order to conceptualize the problem the expert and knowledge
engineer must agree on issues such as:

          o        What are  the  factors involved in decision making?
          o        Which  inputs  give the expert  difficulty?
          o        What are  some of the relationships between  factors?
          o        What factors  are missing?
          o        How accurate  are the factors?
          o        What inferences does the expert make?
          o         How are hypotheses  formed?
          o         How does  the  expert's knowledge evolve?
          o        What factors  suggest particular goals  or concepts?
          o         What are  the  solutions?

          6.6.1  Knowledge Sources

          Several different  [knowledge sources] the knowledge1 engineer can
draw upon while studying  the problem are outlined below.  Sometimes  conflicts
in data will emerge from  different sources.

          6.6.1.1   Background  Data

          The knowledge engineer must be familiar with the domain so as not to
overly burden the  expert  with  questions that can be answered from background
reading.  Background can  come  from various sources including regulations,
guidelines, textbooks, and deliverables or talking with the expert.   Any
policies  and/or procedures that  are already outlined or documented can be a
valuable  source of knowledge.

          6.6.1.2   Cases

          By analyzing specific cases,  the knowledge engineer will be able to
determine what information is  potentially important and how the  factors  are
interrelated.  Cases also give the knowledge engineer an  overall feel for how
the problems were  solved  in  the  past.
                                      6-6

-------
                                                     OSUER Directive #9028.OOa

          6.6.1.3  Test  Data

          Providing test data or posing hypothetical situations to the expert
is another way the knowledge engineer may extract information.  This is done
by observing the way the expert solves the problem with data provided in a
structured test format.  Test data can be derived from historical data
allowing dummy cases to be contrived for purposes of gaining knowledge.


6.7       COLLECTION METHODS

          Given below are various methods for acquiring knowledge about the
domain.  First though, are some general interviewing principles.

          6.7.1	Interviewing Principles

          Points to remember when interviewing an expert include the
following:
          o         The  knowledge  engineer  should be  specific when asking
                    questions.   The  expert  may not have remembered rules or
                    concepts,  and  will  find it difficult  to  recall them.

          o         The  expert should be  encouraged to provide  the information
                    in a way which is most  natural.
                    The expert should not be  interrupted  during unstructured
                    interviews.   The  aim is to  get the  expert talking.
                    Despite the fact  that the expert will probably digress  or
                    repeat things,  interruptions  should be  kept to a minimum.

                    It is important for the knowledge engineer to record all
                    information collected during  an interview and save  it in
                    the Knowledge Notebook.   It is not  always clear which  .
                    parts of the dialogue are important,  even if the questions
                    are planned.   The use of a tape recorder or video  can be
                    very useful if  the expert is  comfortable having the
                    information acquired using  these techniques.

                    The knowledge engineer should listen  to the way the expert
                    uses knowledge.  It is not  just facts,  theories, and
                    heuristic that  are important.  The  knowledge engineer
                    should listen carefully to  the way  in which the expert
                    manipulates knowledge.
                                      6-7

-------
                                                     OSWER Directive #9028.OOa

         6.7.2	Knowledge Acquisition Methodologies

         There  are various proven knowledge acquisition techniques  which
currently exist.  A few are described below.

         6.7.2.1   Unstructured Interview

         An unstructured interview  is basically  a  free form interview in
         which  the expert talks  freely  about  the domain of expertise.  The
         goal is  to  become comfortable  with the  expert and to see how the
         expert views the problem domain.  The unstructured interview is not
         very efficient,  but this technique can  be used to address  several
         important questions:

         o        Who are the experts?

         o        Is the problem scope suitable for an expert  system?

         o        What are  the needs of the  end users?


         6.7.2.2   Structured Interview

         The structured interview is the most basic knowledge collection
         technique used by the knowledge engineer.  -It  is generally used
         after  unstructured interviews  are conducted.  This method is
        . applicable  to all types of problems  and with all types of experts.
         After  the knowledge engineer has  met with the expert several times
         and structured some of  the knowledge,  the knowledge engineer then
         tailors  the interview around gaps in the  information.   Interview
         questions  should center on the expert's knowledge of factors,
         relationships,  and inferences.

         6.7.2.3   Observation

         In some  cases,  the  best way to discover how an  expert makes a
         judgment,  diagnosis, or design decision is to watch the expert work.
         This method is used primarily  when  the  task at hand involves more
         than just  a cognitive process  to  achieve  a goal.  For instance,
         determining which equations the expert  refers  to most often while
         solving  a problem.   The knowledge engineer may  discover the
         knowledge  in various ways.

         One way  is  to watch, take  notes and  follow the  expert's thinking
         processes.   The drawbacks  to this are  that the process is time
         consuming  and that  the  expert  may perform differently while being
         observed.

         A second method is  to record the  session.  This  can be advantageous
         because  it:

                                      6-8

-------
                                            OSUER Directive «9028.00a


o         Retains all information from the interview,

o         Can keep pace with the expert, and

o         Keeps the knowledge engineer's attention on the interview
          and not on recording the information.

The drawbacks  to  this method are:

o         All information must be transcribed,

o         Irrelevant pieces must be filtered out,

o         There is nothing to refer to during the interview, and

o         The expert may be intimidated by a recording device.

6.7.2.4   Decision Matrix

Here  the knowledge  engineer forms  a table or  matrix  with  the
decisions  to be made along one  axis and the different  factors  that
go into making these decisions  on  the  other.   This method is used
when  many  factors are borne out during interviewing  and must be
organized  in some fashion.  The purpose is to better diagram  the
decision making process  and to  identify gaps  in  the  knowledge  for
future interviews.

6.7.2.5   Interruption Analysis

Interruption Analysis  is especially good for  validation of the
reasoning  processes already encoded in the expert system
application.   It  works  in conjunction  with the observation technique
described  in 6.7.2.3.   In this  method  the knowledge  engineer
observes while the  expert proceeds to  solve a problem  without
verbalizing the process.  When  the process gets  to the point  that
the observer can  no longer understand.the expert's actions, the
knowledge  engineer  interrupts.   The knowledge engineer then probes
deeper, asking the  expert about what was happening at  that
particular instant. Note that  once the process  is interrupted there
is a  chance that  it cannot be started  up again.

6.7.2.6   Constrained-Processing Tasks

In this method the  expert is  requested to solve  the  problem in a
much  shortened time frame or  with limited information.  Experts are
.generally  uncomfortable with  limited  information or  time  available
to solve  the problem.   Some experts would rather give  no  answer at
all than  give  one based on incomplete  information.   Explain to the
expert that this  is not a test  of their expertise but  rather  a means

                             6-9

-------
                                                     OSUER Directive #9028.OOa

          of extracting additional information about the domain.   Once the
          expert opens up and becomes more comfortable about giving uncertain
          or qualified judgments,  much can be learned about the experts's
          reasoning processes.  There are two types of constrained processing
          tasks.

          The first is a constrained time task.  Constrained time tasks limit
          the time an expert has to solve the problem.  When a time constraint
          is imposed, the expert will generally take the most direct path to
          the solution.  This method is used to eliminate extraneous thought
          processes which may be occurring without the expert realizing them.

          The second is to limit the information available to the expert.
          This is a good way to determine which factors or data are most
          significant by forcing the expert to rely heavily on their knowledge
          and reasoning skills.  Formulation of hypothesis, use of heuristic,
          and strategic thinking are important in limited information tasks.
          Often, the expert is unaware of factors which are most important and
          this is an excellent way of finding out.


 6.8       RESOLVING CONFLICTS

          As the knowledge engineer gains more and more knowledge about the
 domain,  conflicting.information builds.   Listed below are  some techniques used
 to resolve  these conflicts.
i
          6.8.1 Focus Groups

          When there are cases where multiple experts having different domains
 of knowledge are involved,  focus  groups  can be formed to decide on the best
 method of resolving the conflict.

          6.8.2 Delphi Method

          The delphi method consists of writing  out  each piece of conflicting
 information and evaluating each for its  own merit without regard as to its
 source.   Rationale is given for each and an agreement is made as to the best
 answer.

          6.8.3  Consensus

          A general  consensus can be made based  on discussions with multiple
 experts. The disagreement may have been simply the way in which the knowledge
 was  represented or presented.  This method is often best when dealing with
 incomplete  or inconsistent knowledge.
                                      6-10

-------
                                                     OSUER Directive #9028.00«

          6.8.4	Quality Assurance Committee

          The quality assurance committee is set up specifically to resolve
conflicts and maintain knowledge integrity.  This committee should be set up
initially if many conflicts are expected to occur among experts.

          6.8.5    Hierarchically

          Though not the ideal way,  sometimes it is more efficient to resolve
conflicts in a hierarchical fashion.  If people at one level have trouble
resolving a conflict the person higher up in the chain of command such as a
department chief or manager will resolve it.  This method works best early in
the Development Stage where decisions must be made quickly to avoid
unnecessary delays.   If the manager being asked to make the decision is not a
domain expert, reconsider escalating the issue to that level.


6.9       KNOWLEDGE NOTEBOOK^

          The Knowledge Notebook contains notes on all knowledge engineering
sessions with the expert including any knowledge maps the knowledge engineer
has developed and any real or perceived knowledge gaps.  The Knowledge
Notebook serves to organize the knowledge engineer knowledge and prevent the
knowledge engineer from asking the expert redundant information.


6.10      PROTOTYPING STEPS

          Prototyping reaches its final stages during the Development Stage.
Below are descriptions and development methodologies of the Full Prototype and
the Production System produced during the Development Stage.

          6.10.1    Full Prototype

          The Full Prototype contains most of the knowledge and the
representation and control structures have been determined.  Sample cases and
lines of reasoning are expanded upon.  Difficult cases are added and any
external interfaces are connected.  Testing  and verification now play a more
important role as information and lines of reasoning are expanded.

          The purpose of the Full Prototype is to verify the problem domain,
the knowledge representation schemes and control structures, data
requirements, and interfaces.. If major revisions are needed either to the
knowledge representation or control strategies, a new approach  can be designed
.following the guidelines in CHAPTER 4.

          Prototyping is an exploratory process.  If at this point the design
is no longer valid, the prototype can be reevaluated based on the new
information and redesigned  if  appropriate.   The efforts  that went  into the


                                     6-11

-------
                                                     OSWER Directive #9028.OOa

development are not lost; they represent the design phase in conventional
systems.

          6.10.2   Production System

          This is the actual system that will go out into  the field.  At this
point development is complete and the expert system has a finished knowledge -
base which has been compiled and is secure.  Any additions or deletions which
must be performed as a result of further evaluation and testing are
implemented.  All user interfaces and integration with external interfaces are
in place.  In addition, the system has been fully tested and validated.  Any
disclaimers about the use and liability of the expert system are added to this
prototype.  The completed system should be robust and user ready.


6.11      TESTING AND VALIDATION

          Prior to the field testing performed in the Implementation Stage
(see Section 7.6.2), it is  important that the expert system  is tested and
validated by the developers.  This will ensure that the Production System is
working effectively.  The items to be tested  include the data, the knowledge
base, and the flow of logic.

          6.11.1   Data

          Data validation is the most often neglected task in expert system
development.  Databases, external  files, and  tables must be  correct before the
expert can reason using this data.* If  the data  is  incorrect the system will
fail to produce  the  required results.   This  is true of the results to date as
well as any future results.

          6.11.2   Knowledge Base

          The data that results from the knowledge acquisition processes
should be comprised  only of valid  information.   The data should contain truths
about the expert's knowledge and reasoning and  the  expert's  perceptions and
observations on the  domain.  Also, the  data must be complete in the sense that
it covers all of the relevant subdomains of knowledge.

          The knowledge should accurately reflect the way the expert views the
domain and reasons about the task  at hand.  One  way to check this is to see  if
there is  a consensus across experts.  Remember  that some people feel a bias
toward a  computer generated result and  may grade the response more harshly
than one  would  an expert.   To get  around this,  intermix the  computer results
with those of an expert panel without differentiating between the two.
                                     6-12

-------
                                            OSUER Directive 09028.OOa

6.11.3    Logic Flow

There are various methods used to validate  the logic  flow.

o         Run test cases using sample problems to see if the expert
          system arrives at the same conclusion the expert does.

o         Follow chains of reasoning, picking one specific topic
          within the domain and following the reasoning process the
          expert system follows to see if it matches the reasoning
          followed by the expert.

o         Change the firing sequence to see if the expert system
          produces the same results if the firing sequence is
          changed.  If the system fails to respond in the same
          manner,  then the result was a artifact of the control
          structure and not of the knowledge.
                            6-13

-------
                                                     OSUER Directive *9028.00a
                                  CHAPTER 7
                             IMPLEMENTATION STAGE
7.0       INTRODUCTION
         The  Implementation Stage  is  the  second part of  the  Development and
Implementation Phase.  It is the last step before the system is made fully
operational to all of the users in the field.   At this point, the prototype
system has been tested by the expert and the developer, and revisions have
been made.   The system, in its nearly final state, is ready to be tested in
the field by the actual users through the beta test.  The Implementation Stage
includes four main steps:
                                    *
         o        Testing and evaluation of the expert  system by beta  test
                   participants

         o        Revising  the expert system by knowledge engineers

         o        Registering users  and distributing  the system

         o        Training  users  and providing System Documentation.


7.1        OBJECTIVES

         The objectives of the Implementation Stage (Fig.  7.1)  are mainly to
test and finalize the expert system and prepare it for distribution.  This
includes a beta test of the  system by potential users  and a revision of the
system following the test.   In preparation for distribution of the system, a
training program and documentation are completed, and  users are registered.
The expert system itself is  also prepared for distribution through final
debugging and copies of the  run-time version are produced.

7.2       DECISIONS

          A number of important decisions are made during the Implementation
Stage (Fig-. 7.1).  These involve the  testing and revision of the system, as
well'as the distribution of  the system and training of users.' Decisions
involving the beta test include determining who will participate in the test,
and how long the test will last.   The group of beta users should be kept to a
relatively small number in order to limit the volume of  comments on the
system.  However, the beta users should comprise a representative sampling of
the types of, potential users of the system.  An intensive beta test should
generally last 2-3 weeks, or longer if necessary.  This  will give the beta
users adequate time to learn how to operate the system,  apply it to their
work, and evaluate it.

         After beta users comments have been received, the developers of the
expert system must decide what revisions they will make.  This will involve
reviewing the comments and determining how critical the  suggested changes are

                                      7-1

-------
                                                              OSWER DIRECTIVE #9028.00a
                       Deafen
Objectives
        Have beta user* test expert  ayatem  In the field

        Revise  the expert ayatem  If  necessary

        Prepare  training and  documentation  for the ayatem

        Prepare  ayatem for full  distribution
              •  Decisions to be  addressed Include:
                 - who  will  test  the  expert ayatem
                 • how  long  will  the  beta teat  laat
Decisions       *wnat  revisions  will  be  made  to the  ayatem
                 • la  the  expert  ayatem  ready  for distribution
                 - how  will  training   be  Implemented
                 • la  sufficient  funding  available  for  the  remainder
                  of  the  life   cycle
New
Products *
Llat  of  Beta Teat  Participants
Compilation  of
Beta User Comments
Scope of  User  Community
Documentation  of Expert .System
Distribution  Plan and  Schedule
Implementation  Decision  Paper
                                 Figure 7.1
                            Implementation Stage
                    Objectives,  Decisions,  and  Products

-------
                                                     OSUER Directive #9028.OOa

to the success of the expert system.  Project management must agree to the
developer's recommendations of what revisions to make.  There is an important
trade-off between achieving functionality in the system and using resources
efficiently, and this should be considered when determining which revisions
are the most important.  As revisions are implemented, the developers will
decide when the system is ready for final distribution.

          Decisions involving the implementation of a training program are
also made at this time.  Training options are discussed in Section 7.6.4.  As
this stage in the life cycle comes to a close, management staff must determine
that there is sufficient funding for the remaining phases.


7.3       PRODUCTS

          The Implementation Stage results in six products (Fig. 7.1).  These
include:

          o         Distribution Plan and  Schedule
          o         Implementation Decision Paper
          o         Scope of User Community
          o         List of Beta Test Participants
          o         System Documentation
          o         Beta User Comments

          7.3.1     Distribution Plan and  Schedule

          At the beginning of this stage,  it is important to draw up a
Distribution Plan and Schedule,  summarizing the  distribution activities  to be
completed.   Following the  schedule closely is  especially  critical  in this
stage  because of the increased contact  with people  outside of the  system
development staff.

          7.3.2	Implementation Decision Paper
                        *           >
          The Implementation Decision Paper should be composed early in  this
stage.   It should include  all of the decisions to be made during  this stage
 (see  Section 7.2),  and who is responsible for  carrying them  out.   The
Implementation Decision Paper should be approved by the project management
staff to ensure that they are aware of  the decisions and  schedule  that must be
carried out.

          7.3.3	Scope of the User Community

          The final  Scope of the User Community must be established at this
time  in order to plan for  how many systems will  be  distributed and to provide
adequate support once the  system is  shipped to the  field.
                                      7-2

-------
                                                     OSUER Directive #9028.OOa

          7.3.4    List of Beta Test Participants

          Potential beta test participants  are first selected from the group
of end users.  Beta test participants should include those who are interested
in the development of the system and have the time to effectively test the
system.  After contacting the potential beta test participants and locating
those willing to be involved, a List of Beta Test Participants will be
compiled.

          7.3.5	System Documentation

          Documentation of the expert system and instructions for its use are
written during this stage.  It is imp.ortant that these documents are
completed, at least in draft form, prior to the beta test,  in order that they
can also be evaluated.  Further requirements for System Documentation are
covered in Sections 7.4.1 and 7.6.5.

          7.3.6    Beta User Comments

          Following the beta test,  the beta test participants will be
contacted and their comments on the system will be solicited.  These comments
will be integrated and compiled for .use by those involved in the revision of
the expert system.


7.4       SUCCESS FACTORS
                                                                      •
          There are several factors in the  Implementation Stage that must be
considered in order to ensure the success of the expert system.  Most of the
factors involve the transfer of the system to users in the field and the
initial use of the system.  In order to produce a successful expert system, it
is important to:

          o        Provide useful,  readable documentation

          o        Provide sufficient and organized training

          o        Provide sufficient technical support

          o        Ensure  that hardware in  the  field is  adequate  to  support
                   the  expert system  software

          o        Avoid problems with licensing and run-time  versions

          o        Maintain management commitment during the distribution of
                   whe  system.

          Most of these factors can be covered through planning and
coordination prior to this stage.


                                      7-3

-------
                                                     OSUER Directive «9028.00a

          7.4.1	Documentation

          Good documentation can be produced through a quality assurance
system, requiring  reviews by the developers of the system, as well as
management.  A draft of  the documentation must be provided to the beta users
for  their comments on  its clarity  and comprehensiveness.  For more information
on expert system documentation see Section 7.6.5.

          7.4.2	Training

          The planning of the training process should be supervised by
'management, who will work with the developers and the  trainers.  For more
information on training  issues, see  Section 7.6.4.
                                                                             «
          7.4.3	Technical  Support

          In order to provide sufficient technical support, a technical
support team  should be organized before  the. system  is  distributed.  Their
duties  should include  manning a support  hotline  and providing on-line help to
the  users.  The  technical  support  team should include  individuals who were
involved in developing the  expert  system, writing the  documentation, and
providing training sessions.  More information on technical  support for  the
expert  system can  be  found in Section 8.5.2.2.

          7.4.4	Hardware Issues                                    .

          Problems with  hardware in  the  field can be avoided if the users  are
given ample warning of what equipment will be required to run the expert
system.  This process  is explained in Section 7.6.6.

          7.4.5	Licensing Issues                           ,

          Licensing and  run-time issues  should have been  confronted early  in
the  life cycle (s'ee Section 2.5.6.1).  During the Implementation Stage,  it
should  be necessary only to confirm with the vendor of the shell what  steps
should  be taken  to distribute run-time copies of the expert system.  The
matter  of who will pay for the  run-time  copies must also be considered at this
time.  For more  information on  run-time  and licensing issues, see Section
7.6.8.

          7.4.6	Role  of Management

          Management commitment during this stage can be  ensured if, early in
the  life cycle,  management is made aware of the  fact that they  will play a
large role in the  Implementation Stage.   They should know that  they will be
responsible for  the scheduling  and planning  involved in this stage., as well as
the  actual distribution  of the  system and all of the issues  accompanying the
distribution  process.
                                      7-4

-------
                                                     OSUER Directive «9028.00a  '

7.5       ACTIVITIES

          There are several important activities in this stage that lead up to
the distribution of the system.  These include  the collection of data,
testing, demonstrations, and management activities.

          7.5.1    Data Collection

          Collection and organization of data is important during this stage.
Sufficient staff time should be allocated for the following tasks:

          o        Contacting possible users and recording their level of
                   interest .in participating in the beta test

          o        Selecting actual  beta users  and recording their names and
                   addresses on a  mailing list

          o        Seeing  that the beta test system and any accompanying
                   documentation is  sent to all of the  beta users

          o        Developing a questionnaire on the system and sending it  to
                   all of  the beta users or conducting  the survey by  phone

          o        Collecting the  questionnaires and compiling responses and
                   suggestions from  the beta users

          o        Registering all users of the system  and maintaining the
                   registry.

          7.5.2    Testing

          The main testing to be conducted during this  stage  will be the beta
test.  Resources should be allocated for the following  tasks related  to the
beta test:

          o        Contacting beta users  (see Section 7.5.1) before and after
                   the test

          o        Technical support for both hardware  and software

          o        Documentation/training

          o.       Revisions to the  system following the beta test.

          7.5.3	Management  Support

          Management plays an important role in the Implementation Stage,  so
it may be advisable to allocate a  staff member  to serve as an assistant to  the
manager.  This person would provide  administrative support for:


                                      7-5

-------
                                                     OSUER Directive 19028.OOa

         o         The beta test,
         o         Data collection,
         o         Training and documentation,
         o         Licensing and run-time arrangements,
         o         Distribution plans, and
         o         Configuration management.


         7.5.4	Demonstrations and Briefines for the Users

         Contact with the users  is especially important during the
Implementation Stage, as the expert system is nearing its final state.  The
users will be able to see the actual system they will be working with, rather
than a prototype.  Demonstrations and briefings involving possible users can
be useful in determining who is interested in participating in the beta test
and final evaluation of the system.  They will also determine the overall
level of interest in the system.  This information can be used to define the
size of the system, and to allocate sufficient staff time for distribution,
technical support, and maintenance of the system.

         Demonstrations and briefings can also be used as training  sessions
for both the beta users and initial system users.  In some cases, the
developers of the system may choose to meet with some of the beta users after
the test to receive their comments on the system.  This would allow for a more
interactive evaluation than simply filling out a questionnaire.

          7.5.5	On-going Progress Reporting  to Management

         Management should receive a complete schedule of all tasks involved
in implementation at the beginning of the stage.  They  should immediately be
informed of any  changes in the schedule, because many aspects of system
implementation involve outside parties, and maintaining good public relations
is very important.  Progress reports should include updates on:
        •
          o        Beta test,
          o        Distribution  plans, and
          o        Any major problems encountered.


7.6       DISTRIBUTION STRATEGY

         An important part of the Implementation Stage is planning and
scheduling the distribution of the system.  This stage  of implementation has a
number of parts  and should include close management involvement.

          7.6.1	Rollout Plans and  Schedule

         At the beginning of the Implementation Stage, all tasks to be
performed must be carefully planned and scheduled.  Specific dates should be
set for the beginning  and end of  the beta test, and for the distribution of

                                      7-6'

-------
                                                     OSUER Directive #9028.OOa

the final system.  All aspects of the stage should be planned carefully, and
sufficient staff and time should be allocated for each task.  It is very
important that the beta test and distribution of the final system are carried
out smoothly and that the schedule is closely followed, because this stage
will be receiving a great deal of external attention, and public relations is
an important factor.

          7.6.2    Beta Test

          The beta test is a very important step in the life cycle  of an
expert system.  During this step, the system will be distributed to a few
selected participants who have agreed to try the system out for a defined
period of time, then provide their comments on the system and any suggestions
they have for changes or improvements.  The aspects of the system to be tested
include:

          o        Hardware,
          o        Software,
          o        Support procedures,
          o        Documentation, and
          o        Training.

          The beta test provides an opportunity for the system to be tested on
[real-world problems] by the people who will be using the system.  They will
know best if the system will be effective and assist them in their work...

          7.6.2.1  Contact Beta Users

          Potential beta users should be contacted early in the Implementation
          Stage to determine their interest in participating in the test.   A
          small number of interested parties will be selected to participate
          in the beta test.   These participants will receive the entire expert
          system,  along with draft documentation on the system.   The beta
          users can also be provided with training on the use of the system,
          if necessary.   This would provide an opportunity  for the  training
          materials to be tested,  as well.

          7.6.2.2  Register Beta Users

          The beta users should be registered before they begin testing the
          software.  This will make it easier to identify unregistered beta
          users and explain to them the risks of using beta test software.
          The distributors'  risk and exposure will then be  limited if the
          unregistered persons decide to use the system anyway.

          7.6.2.3  Collect Comments from Beta Users

          After approximately 2-3 weeks, or longer if necessary, the beta
          users will be required to answer some questions about the expert
          system and provide comments and suggestions for changes or

                                      7-7

-------
                                                     OSUER Directive #9028.OOa
         improvements.  The beta users may be  sent  a questionnaire, which
         they will have to complete  and return,  or  they may be  contacted by
         phone  and asked  the  questions directly.  The  developers may  also
         wish to meet with some  or all of the  beta  users  and  conduct  an open
         forum  to discuss the expert system.

         7.6.2.4  Revise  Expert  System

         After  the beta users responses and comments have been compiled and
         evaluated,  the developers will revise the  expert system accordingly
         and prepare a final  version for distribution  to  all  users.

         7.6.3     Registered Users
         ™""^"""^^"••^"•^•" - T-J     ~"~^^                             »

         Registration  involves keeping a list  of all authorized system users
and assigning each with a registration number.   This process will be
supervised by the management  support staff.  .All users of the final expert
system should be registered.  However, if there will only be a  small number of
users, this may not be necessary.

         Registration  ensures that all of  the  users are using the  official
run-time version and that unauthorized users will not be supported.  Each user
will be assigned a registration  number, which will be activated when the user
returns the registration  card included with the  software.   The  registration
number must be  stated before  any information or  technical support will be
provided to the user.

         A list" of all registered users  and their addresses  and phone numbers
will be maintained so that they  can be informed  of  any errors in the system or
documentation,  and be provided with new versions of the system  when they are
developed (see  Section 7.6.9).

          7.6.4    Training Issues

         Training of all system users is an important part: of ensuring that
the expert system is operated and  maintained properly  in  the  field.  Users can
be trained directly through face-to-face training sessions,, or  indirectly
through other training methods.

          7.6.4.1  Direct Training

          Direct training of the users can reduce the amount of technical
          support necessary and can make  the transition into the initial
          operation of the expert system much smoother. However,  direct
          training may not be necessary or desirable,  if,  for example:

          o        The users  are widely scattered geographically,

          o        There  is a large  number  of  users,


                                      7-8

-------
                                                     OSUER Directive 09028.OOa

          o        The system is self-explanatory, and/or

          o        The majority of the users have seen demonstrations of the
                   expert system and have a good understanding of its
                   operation.


          If direct training is  implemented, those performing the training
          sessions  should be persons who were  directly  involved  in the
          development process, such as  the knowledge  engineers,  or instructors
          who have  been trained  directly by  them.  Training sessions  could be
          at the users'  location, or in a central location,  with one  or more
          sessions.

          It should be determined if every user will  be involved in the
          training,  or if there  will be a single representative  or small group
          from each area or  region.  Training  a small number of
          representatives is known as a "train-the-trainers" approach.  After
          these individuals  have been trained, they will be able to train the
          other users in their own area or region.

          7.6.4.2  Other Training Methods

          If direct training is 'not implemented, several other methods of
          training  can be used.  Training can  be put  on-line as  part  of the
          expert system,  it  can  be recorded  on videotape and sent to  the
          users,  or it could be  provided in  the form  of a written training
          manual.

          It may be useful to provide training at several levels.   The initial
          set of training could  be a beginning level, and could  be followed by
          more advanced training, after users  have some experience with the
          system.

          7.6.4.3  Intended  Purpose of  the Expert System

          An 'important part  of the training  will be emphasizing  the intended
          use of the expert  system.  An expert system is meant to be  advisory
          in nature,  and will not take  the place of a human.  The users will
          receive recommendations from  the expert system, but it remains their
          responsibility to  make all final decisions, either accepting or
          rejecting the system's recommendations.

          7.6.5	Documentation

          Document, tion of the expert system is very  important and is
something that is often inadequate.   The more clear and easily understandable
the documentation is, the less technical support will be needed.  The
documentation should probably not be written by someone who was directly
involved in the development of the expert system,  such as a knowledge engineer

                                     7-9

-------
                                                     OSUER Directive #9028.OOa

or programmer.  They may leave out details that are obvious to them but not to
a first time user.

          The developers  of the  system  should be  involved in a review  of the
documentation, but it may be preferable to have the documentation written by
someone who is trained by the developers, but who has Just become familiar
with the system.  Such a person will be more aware of every necessary step and
detail involved in running the system.   Another option would be to use a
technical writer, who could study the system and be briefed by its developers,
then write the documentation.

          The documentation should be brief,  clear,  concise, and  use graphics
and pictures of the actual screens whenever possible.  No system will be
shipped without adequate documentation.

          7.6.6	Hardware Requirements

          Prior to distribution of the  final expert system,  all users  will
need to have the hardware necessary  to support the expert system.  Early in
the Implementation Stage, all potential users should be informed of the
hardware, equipment, and facilities  they will need to run the  system, such as:

          o        Type  of computer, amount  of memory, whether a hard disk is
                   necessary

          o        Type  of furniture and work space  needed to accommodate the
                   computer
                                           •
          o        Type  of printers

          o        Phone lines  and modems

          o        Electrical hookups  and surge  suppressor.

          Ideally, the hardware selection process, described in Section 4.11
took into account the  facilities  of  the majority of  the potential users, and
the target hardware was  chosen  accordingly.

          7.6.7	Software Requirements

          Prior to distribution of the final expert system, the software must
be prepared  for the users.  A run-time copy  must be  prepared  for each user and
labeled.  Everything necessary  for the system should be packaged together and
sent to all  users with detailed instructions.

          7.6.8	Run-time and Licensing Issues

          Before the final expert system is distributed,  all run-time and
Licensing issues  must  be resolved.   Although the major decision point on this
subject occurred  when  the  development  and delivery environments were  evaluated

                                     7-10

-------
                                                     OSUER Directive #9028.00a

in the Definition and Design Phase  (Section 5.5), these  issues must be
addressed in the Implementation Stage.  All user facilities will receive a
run-time version of the software.   Some of the expert system shells currently
on the market require a fee for each run-time copy issued, while others have a
one time fee that covers all run-time copies.  If the run-time fees are to be
allocated to the users, a system for the collection of fees must be
implemented.

          Some expert system shells do not include run-time modules.   These
shells do not provide any protection for the expert system, and would allow
the users to make changes to it.  If this is the case, a disclaimer must be
included, stating that the system will no longer be valid if it is tampered
with or changed in any way.

          The final version of the expert system should include the copyright
information for the company that owns the rights to the expert system shell,
as well as a notification that it is illegal to make copies of the software.

          7.6.9	Configuration Management and Version  Control

          When the expert system is revised or a new version is developed,  it
will be necessary to distribute it  to all users.  Users must have the
currently valid version, and only that version, to avoid confusion and legal
infractions of the licensing agreement.  When a new version becomes available,
all users will be notified.  They will be required to send in their old
version, which is marked with their registration number, in order to receive
the new one.  Other matters to be considered when the system is revised
include:

          o        Changes or addendum to the documentation

          o        Retraining

          o        Removal or overwriting of old versions from all of the
                   users' hard disks.                    .         :
                                     7-11

-------
                                                     OSUER Directive 49028.OOa
                                  CHAPTER  8
                                OPERATION PHASE
8.0       INTRODUCTION
          After completing all four major phases (Initiation,  Concept,
Definition and Design, and Development and Implementation),  Operation is the
last of the five major phases in the OSWER system life cycle.  This phase is
divided into three stages, Production, Evaluation, and Archive.

          The Production Stage starts with the results of the work conducted
in all prior stages, and activities of this stage are frequently in response
to those results.  In other words, deficiencies and problems not resolved
adequately in prior stages are usually identified by users during the
Production Stage.

          The Evaluation Stage differs from .other phases and stages of the
system life cycle in that  it occurs simultaneously with another stage
(Production) and may be repeated more than once during the system's
operational life.  During  this stage, the system is reviewed formally to
determine whether it is operating  correctly and efficiently from a technical
standpoint, and whether it continues to  effectively address the information
management problem.

          The Archive Stage preserves information not only about the current
production system, but also about  the evolution of the system through its life
cycle.


8.1       OBJECTIVES

          Figure 8.1 graphically illustrates the objectives  associated with
the Operation  Phase. • Each stage's objectives are cited below.

          8.1.1     Production

          The first objective of the Production Stage is to  use the
capabilities of the system to solve the  information management problem by
delivering the system to the users.  Proper use of the system will require
user training  in the special capabilities and limitations of expert systems.

          The second objective is to identify potential changes needed to
ensure that the system and data continue to solve the information management
problem. • Changes may take the form of routine maintenance or may constitute
enhancements to the system or databases.  Careful management of the knowledge
base, including revalidation after each  change, will be required to ensure
continued satisfactory performance.
                                      8-1

-------
                                                            OSWER DIRECTIVE #9028.000
Objectives
Decisions
Production:
 • deliver the  system to  the  users
 • Identify  potentlsl changes  to the  system
 - develop  snd  Implement  msintensnee changes  end minor
  enhancements

Evaluation:
 • determine the extent  to which  the system effectively  snd
  efficiently addresses  the  Informstlon  msnsgement  problem
 »determine whether  the' system  should  be  continued,
  enhanced,   or  replaced/archived

Archive:
 - end the  operation of  the system In sn  orderly  msnner
 - ensure system components  snd  dsts  sre  properly  archived
  or  Incorporated  into  other systems
Production approsch  decisions include:
 • what evaluations  of  the  system  snd dsta should  be  conducted
 - what new  or  additional  user' support activities  are  needed

Production execution decisions  Include:
 - what changes or  enhancements are  needed
 - should  sn  enhancement be  made within  this  stage or
  given  Ita  own  life cycle

Evaluation  approach decisions Include:
 • how well does  the  expert  system solve the Information
  management  problem
New         •  Performance  Report
Products    •  Post Implementation Evaluation
                Report
                                   • System Evaluation Report
                                   • System Disposition  Report
                                 Figure  8.1
                               Operation  Phase
                     Objectives,  Decisions, and  Products

-------
                                                     OSWER Directive #9028.00«
          The  third objective is  to develop and implement maintenance  changes
and minor enhancements.  All maintenance and minor changes are controlled
through baselines, particularly in the case of expert systems.
                                                                      i>
          8.1.2	Evaluation-      •                                  #
                                                                      i;
          The  first objective of the Evaluation Stage is to determine  the
extent to which the system effectively and efficiently addresses the
information management problem.  Efficient operation, effective management,
and current functional and data requirements are all considered in this  stage.

          The  second objective is to determine whether the system should be
continued, enhanced or replaced/archived.  This decision  requires appraisal  of
advances  in technology and estimation of cost/benefit to  upgrade the  system.

          8.1.3     Archive

          The  first objective of "the Archive'Stage is to end the operation of
the system in a planned orderly manner.  Care must be taken not to disrupt
OSWER programs that use the  system as well as other  systems that use  the data
and/or software of the current system.

          The  second objective is to ensure that system components and data
are properly  archived or  incorporated into other, systems.  Resources  dedicated
to the system-being deactivated  should be  reallocated to  functioning  systems
where appropriate.  Data  and software that- are  not currently  needed by other
systems should be archived against future needs.


8.2       DECISIONS

          Many of the decisions made during this phase of expert system
development are identical to those made  for  conventional  systems.  Only
special expert system  development  considerations are cited below.  If
additional detail is required, refer to  the  LCM Guidance.

          8.2.1     Production

          8.2.1.1   Approach  Decisions

          The Production  Stage deals with two approach decisions.  First, what
          evaluations of  the system and data should be conducted?  Many
          factors may affect the selection of specific evaluations including
          new requirements,  system performance, new technology, and cost
          experience (i.e.,  costs accrued over the development life cycle).
          Because expert  system technology in particular  is subject to rapid
          evolution, system developers must stay abreast  of advances  in AI
          tools and techniques.
                                       8-2

-------
                                                     OSUER Directive *W28.00a

         Second, what new or  additional user support activities are needed?
         Much will be learned about  the required level of  system maintenance .
         and user training as the  system  is developed.  Because improper use
         of  an expert system  could have serious repercussions,  these topics
         should be monitored.carefully to ensure that the  systeia is used
         properly.                                              ;;

         8.2.1.2   Execution Decisions

         There are two  execution decisions.  First, it must be  determined
         what changes and/or  enhancements to the system and database are
         needed.

         Second, should a particular enhancement be implemented within this
         stage, or given its  own life cycle?  Potential factors to  be
         considered which tend  to  require the start of a new life cycle
         include:

         o         The processing of additional information,

         o         Major impacts on the system,

         o         High cost of accomplishing the change,  or

         o         Significant impact on multiple OSVER,  regional,  or state
                   offices.

         Review and approval  thresholds will play a major  role  in determining
         what action  to take.   The capacity of the expert  system software to
         be  modified  will influence  this  decision.

         8.2.2     Evaluation

         The Evaluation Stage deals  with  what changes or improvements in
         system and data functionality, quality, and/or performance are •
         needed.  This  approach decision  made during this  stage will
         determine how  well the system solves the information management
         problem.  Information  access, processing, and content  are  all
         considered.  Expert  systems will require special  attention in
         evaluating changes to  the knowledge embedded within the system
         (e.g., as regulations  or  allowance -changes).
8.3       SUCCESS  FACTORS

          Several  factors  that can impede  success  it  not considered in the
Production, Evaluation, and Archive Stages are described below.  This Section
focuses on perceptions of what expert systems can and cannot do and offers
solutions or alternatives.
                                      8-3

-------
                                                     OSWER Directive #9028.OOa
          8.3.1     Production

          In the  Production Stage,  users  should not become  overly reliant on
an expert system's recommendations.  Because the advice offered by the system
is so similar to the human expert's recommendations,  the advice may be
accepted as infallible.  This is particularly dangerous when the system is
working with incomplete or incorrect information.  Users should maintain an
attitude of skepticism in proportion to the consequences of the system
rendering incorrect advice.

          8.3.2     Evaluation

          In the  Evaluation Stage,  the organization should be  cognizant of
changes that affect the system's performance and should not treat them too
casually.  Relevant changes such as development of new techniques or
expertise, and enhancements in hardware and software may all affect the
system.  Formal evaluations of all perceived changes and their impact on the
system are required.

          Translating changes into modifications to the knowledge base will
require an iteration of.knowledge  engineering with the expert.  Depending on
the extent of the changes, this iteration could almost be  a mini-project
development effort.

          Over time,  the problem that the system was  developed to address may
cease to exist or may be subsumed  in a larger problem addressed by another
system.  "Pride of ownership" can  inhibit shutting down a  system  that is no
longer required.  This is  obviously wasteful and inefficient.  All programs
and systems should be viewed objectively.

          8.3.3    Archive

          In the Archive Stage,  it is important to retain components of the
system that are useful at  a later  date.  In general, knowledge is always
valuable.  Knowledge in automated  form is easily archived  and retrieved  as
needed.  However, it is also easy  to overcompensate for the first by retaining
everything.  If an information management problem ceases to exist, significant
portions of the solution to the problem are probably discardable.  Large
systems may consume considerable physical and  logical storage space; both are
expensive and should be used efficiently.


8.4       PRODUCTS

          Figure 8.1 also graphically illustrates the new products associated
with the Operation Phase.  They are similar to  the ones cited in  the LCM
Guidance.  Each stage's products are listed below.
                                      8-4

-------
                                                     OSUER Directive #9028.OOa

          8.4.1	Performance Report

          The Performance Report describes the experience of system, knowledge
base, and database users during Production, noting unanticipated events and
potential problems.  This report serves as a diagnostic ;;ool to aid the
project manager, as well as assisting evaluations of the;system, knowledge
base and database(s).  This report is usually brief, and may include extracts
from computer facility reports that identify the resources used by  the system
and database(s).

          8.4.2     Evaluation

          8.4.2.1   Post-Implementation Evaluation Report

          The Post-Implementation Evaluation Report presents a complete
          assessment of the implemented system based on the experience of the
          initial period of system operation.   This report  addresses all
          facets of the system,  including degree of satisfaction of functional
          and data requirements,  technical performance, and system management.
          It also identifies potential  new requirements not addressed by the
          system.  Specific recommendations are provided, where appropriate,
          to help ensure  that the system continues  to respond to the
          identified information management problem.   With  regard to expert
          systems,  particular attention should be paid to new or changing
          functional and data requirements, ongoing training,  and database and
          knowledge base  management.

          8.4.2.2   System Evaluation Report

          The System Evaluation Report  presents the results of a formal
          assessment of the system.  'The assessment may vary in scope,
          focusing on how well the system addresses the information management
          problem,  technical performance of the system, and/or system
          management practices.   The report provides specific recommendations
          where appropriate and notes those recommendations approved by
          program management for action.   If the evaluation is conducted by a
          completely independent third party,  the evaluation should include a
          Section including the opinion of the project team with regard to the
          findings and recommendations  of the evaluation.

          8.4.3     System Disposition Report

          The System Disposition Report:

          o         Describes  the  rationale for ceasing system  operations,

          o         Documents  the  plan  for ceasing  operations  and effectively
                    archiving  the  various  components  of the system,  and
                                      8-5

-------
                                                     OSWER Directive #9028.OOa

          o        Provides information about the location of archived
                   materials.

          This report is vital to ensure  that information about the system can
be accessed to support reactivation of the system, or future reuse of portions
of the current system by other systems.
8.5       ACTIVITIES

          In addition to activities resulting in the creation of the products
listed in Section 8.4,  the project management and development team  should
conduct activities  to resolve human resource issues, ensure  the  successful
completion of the production stage, maintain the system, and evaluate  the
system.  These activities are described in the following Sections.

          8.5.1	Human Resource Activities

          8.5.1.1   Soliciting  Feedback from  the Users

          The users of the expert system play the most important role in the
          Production and Evaluation Stages.  By this time the expert system is
          a fully working,  usable system and the user can thoroughly critique
          its performance.   They are best able to determine  if the system is
          efficient and effective.. It is important to gather the user's
          comments and recommendations at this phase to evaluate if the system
          should be continued,-enhanced,  or archived.

          8.5.1.2   Two-Wav  Communication  with  Management

          Management also has a heavy involvement in these stages.  If the
          users desire to continue using the system, they must have
          management's support to allocate funding and resources.  Likewise,
          management must evaluate the user's requests and comments and
          incorporate them so that the system maintains its  effectiveness.

          8.5.2 Production Activities
                                                             /
          Project managers and software developers must also consider
          distribution and end-user support for the completed system.  The
          distribution issues are presented in Section 8.5.2.1;  the end-user
          support issues in Section 8.5.2.2.

          8.5.2.1   Distribution

                    8.5.2.1.1	Configuration Management

                    Formal  review of requested  changes  to the system before
                    they are made is essential  to  the integrity of the system.


                                       8-6

-------
                                            OSWER Directive #9023.00*

          The procedures for reviewing requested changes are
          contained in the Configuration Management Flan (a part of .
          the overall Froj ect Management Plan).

          8.5.2.1.2	Software Disclaimers
                                        \>
          Software disclaimers use limited warranties to restrict
          vendor and manufacturer liability to replacement of faulty
          software.  They typically deny responsibility for the
          consequences of improper use of the software or for use of
          outdated versions.  Because the advisory capacity of
          expert systems presents new implications for improper
          system development or use,  any software disclaimers by
          vendors,  manufacturers, or developers  should be carefully
          studied.   Standard disclaimers will probably emerge in the
          near future as expert systems applications become "off-
          the-shelf" products.

8.5.2.2   End-User Support Requirements

          8.5.2.2.1	Ongoing Training

          Providing ongoing training is important in the Production
          Phase.   New users will require training to effectively use
          the system, and experienced users will require retraining
          whenever significant changes are made  to the system.
          Systems that generate critical output  may require regular
          certification of all users.

          8.5.2.2.2	Documentation
                                                     i
          All changes should be completely documented to provide
          system users and those responsible for operating and
          maintaining the system the information they need to
          properly use the system and to.perform -other activities
          for effective system use.
                                      t
          Effective reactivation of the system in the future will
          depend heavily on having complete documentation.  It is
          generally advisable to 'archive all documentation,
          including the life cycle products generated during the
          earliest stages of the life cycle, as  well as the
          documentation for users and for operation and maintenance
          personnel.

          8.5.2.2.3	User Groups

          Manufacturers of larger expert system shells support user
          groups in their major client bases.  These groups provide
                             8-7

-------
                                                     OSUER Directive #9028.OOa

                   a forum for exchanging experiences with the tool and with
                   particular applications of it.  All organizations should
                   be represented at user group meetings.

                   8.5.2.2.4	User Feedback

                   User feedback helps to determine what enhancements to the
                   system are needed to continue to solve the information
                   management problem, including requested changes deferred
                   from prior stages, and developing and implementing these
                   enhancements.

                   8.5.2.2.5	Software Updates and Upgrades

                   Software updates and upgrades are necessary in the
                   Production Phase to ensure a successful evaluation for the
                   continued use of the system.  Any new releases of system
                   software and applications software packages to operate the
                   system should be provided.

          8.5.3    Evaluation Activities

          Evaluation considerations  that  must be  considered by the  project
manager or software developer are maintenance for the system and a post
implementation evaluation..  These two considerations are described in the
following Sections.

          8.5.3.1  Maintenance

          During the Evaluation Stage,  potential  changes  to the system are
          considered to ensure  that  the system  continues  to solve  the
          information management problem.   Changes  may take the form of
          routine maintenance or may constitute formal enhancements to the
          system or databases.

 1                  8.5.3.1.1          Maintenance  and Revalidation Options

                   There are three major sources of change proposals and
                   maintenance activities:  users, experts, and the MIS
                   department.

                   o      •   Users typically request minor enhancements that
                             provide additional functionality and/or improve
                             performance  but do not alter the knowledge or
                             data structures.  They provide maintenance
                             support by putting any changes  through "real
                             world"  testing.   Experienced'user criticisms and
                             comments should be considered carefully:  they
                             know best how the  system works and does not work.


                                      8-8          .

-------
                                            OSUER Directive *9028.00a
          o        After an expert  system has been developed and
                   fielded, the problem domain may change for
                   technical, regulatory, or procedural reasons.
                   The experts wj11 probably be the first to
                   recognize the impact of these changes on the
                   system.  They should therefore be regularly
                   involved in assessing how well the system
                   continues to address the problem.  They will also
                   assist in revalidating the system after any
                   change is implemented by evaluating the effects
                   of the change on other parts of the system.

          o        In most organizations, the ADF department is
                   responsible for  actually implementing changes to
                   supported systems.  Just as in system
                   development, MIS staff will need to understand
                   expert system concepts in order to properly test
                   and validate after making changes.  They are also
                   an excellent source of knowledge about system
                   capacity and load patterns.

          8.5.3.1.2    	Knowledge Maintenance

          If the system has  captured all the  relevant knowledge  in
          the task domain and if  that knowledge will not change  in
          the foreseeable future,  then knowledge maintenance will
          not be needed.   Few problem domains, however, are  totally
          static.   If an  expert system has been developed
          specifically because  the  task domain is  changing,  then
          users, experts,  and the MIS department will all
          participate in  a prolonged Production Phase that will
          incorporate multiple  Evaluation  Stages.

8.5.3.2   Post-Implementation Evaluation   •                    j

The system should be reviewed formally to determine whether it ,is
operating correctly and efficiently from a technical standpoint, and
whether it continues to effectively address the information
management problem.  This'formal Post Implementation Evaluation is
conducted only  once, during the first occurrence of the Evaluation
Stage.  The project manager or software developer should evaluate
user satisfaction and task performance as well as prepare a benefit
analysis during the Post-Implementation Evaluation.

          8.5.3.2.1	User Satisfaction

          The degree of user satisfaction  will be  reflected  in the
          Post-Implementation Report.                             ,  .
                             8-9

-------
                                  OSUER Directive #9028.OOa
8.5.3.2.2	Task Performance

The system should be evaluated for task performances
against the system specifications.  The Post-
Implementation Evaluation is used to perform a
comprehensive appraisal of the system.  This evaluation
addresses all tasks of the system:  support of functional
and data requirements, system technical performance, and
effectiveness of system management.  All specialized
expert system test, review, and validation procedures used
in developing the system are now used to evaluate it.

8.5.3.2.3	Benefits Analysis

For each evaluation in this stage a decision is made
regarding the future of the system.  If it is operating
correctly and effectively, the Production Stage continues.
This decision requires appraisal of advances in technology
and estimation of cost/benefit to upgrade the system.  If
the system is no longer operating correctly or
effectively, and improvements would not be cost-effective,
the life cycle proceeds to the Archive Stage.
                   8-10

-------
                                            OSWER Directive #9028.OOa
                            APPENDIX A
           EVALUATING RULE-BASED EXPERT SYSTEM SHELLS
A.I  INTRODUCTION            ;•

        This appendix describes a set of criteria for evaluating
and comparing rule-based expert system development environments
(or shells).  These guidelines will assist expert system
developers in making an informed, objective decision when
selecting a rule-based shell that is well-suited to their
development objectives and requirements.

        Commercially available expert system shells are
proliferating; every major and minor software developer seemingly
is in the market.  Prices of the various packages range from less
than $100 to almost $100,000 — the majority are sold for less
than $5,000.  The capabilities and quality of the shells cover
almost as broad a range, but not in direct proportion to price.
With the constant emergence of new packages and frequent upgrades
to older ones, the market is extremely volatile; buyers must
beware of poorly supported or otherwise inferior software.  Since
there are no standards against which to evaluate a particular
shell and different shells are best suited to different types of
problems, a strategy must be developed to address the problem.

        The evaluation of. expert system shells differs from that
of languages and other programming tools because of the lack of
standards.  The challenge for the system developer is to
carefully consider the characteristics of his project and then
match those -characteristics to a shell that is appropriate.   If
the type of problem to be addressed by the shell is known, then a
small benchmark rule base can be designed for implementation on
the various shells being evaluated.  In this way a standard of
comparison can be loosely set.  If the problem type is unknown or
varied, then the shell must be evaluated in a more abstract
fashion.

        This appendix describes the major factors that should be
considered as part of a comprehensive evaluation of rule-based
expert system shells.  Two attachments accompany this appendix:
the first is an evaluation form that incorporates the major
evaluation factors in a convenient checklist; and the second is a
comparison of expert system application types to shell
characteristics that is included to aid the developer in
determining what type of shell is best suited to a given problem.
The conclusion provides some general comments and recommendations
to guide expert system developers in the evaluation process.
                                A-l

-------
                                            OSUER Directive #9028.OOa
A.2     SHELL EVALUATION FACTORS

        This Section briefly describes the important factors that
should be considered in the comprehensive evaluation of a rule-
based expert system shell.  These factors have also been listed
in Attachment A.I, Expert System Shell Evaluation Form.  In
evaluating a particular shell, the evaluator should quantify
whatever values possible in order to enhance the objectivity of
the evaluation.  The descriptions below attempt to summarize the
types of issues that should be addressed in each Section of the
form.

        A.2.1  General Information

        Attachment A.I, Sections 1 through 4 provide an outline
for recording general information.  Since the shells being
considered may be reviewed by more than one person or by persons
unknown to each other, record should' be kept of the evaluators.
Vendor contacts should be listed in order to provide a source  for
further purchase or evaluation information.  Any impressions of
the vendor should also be noted, such as how long the shell has
been on the market or general impressions about the quality of
the vendor's product line.  Hardware and software requirements
should be noted also.  Attention should be paid to the minimal
and recommended configuration and the actual configuration used
to test the shell.

        A.2.2  Features

        A survey of the features present in the shell is an
important initial step in assessing the shell.  The key features
that should be evaluated are described below.  These features  are
listed in Section 5 of Attachment A.I.             .      .

        A.2.2.1  Structure   (Attachment A.I, Section 5.1)

        Different software tools use different types of
        parameters to represent values.  Many, such as logical
        and string variables, are commonly supported by expert
        system shells, but some common ones, such as arrays and
        floating point numbers, frequently are not.  In order  to
        determine which data types are required, the evaluator
        must fully define the problem domain in advance.  If that
        is impossible, then all data types that may possibly be
        needed should be considered.

        Flexibility in the formation of rules varies widely among
        shells.  Some do not permit multiple clauses in the


                                A-2

-------
                                    OSUER Directive #9028.00a

premise  (IF) Section of the rule, or make allowance for
an alternate conclusion (ELSE, ELSE IF).  Once again, the
particulars of the problem to be addressed will dictate
what rule structure is needed, but greater flexibility
will permit more general application of the package.

Many expert systems must be capable of handling uncertain
or incomplete data.  The method used by a particular
shell to address uncertainty is an issue of concern.  How
does the shell treat confidence factors (if at all)?
Does the shell permit probabilities only in the rule
premise?  What formula does it use to combine
probabilities?  What scales (e.g., -1 to +1, 0 to 100)
are available?

Some applications may require extensive mathematical
capabilities.  The accuracy of mathematical calculations
and comparisons will vary depending on the internal
representation method; a variable may display a value of
OcOOO but still not act as a zero flag because its actual
value is 0.0001.

Frequently expert systems rely on external software to
perform data calls and mathematical operations.  A
desirable shell feature is the ability to link to popular
applications packages such as 1-2-3-or dBASE III, or
access routines written in programming languages such as
C, Pascal, or assembler.  Another consideration is
whether the shell can access the underlying operating
system and utilities such as the system clock.

The type of problem to be addressed generally determines
the type of knowledge representation to be used.  There
are problems that may be addressed in several ways and
the differences between the methods of knowledge
representation are sometimes subtle.   Rules, frames, and
scripts are the most widely .used representation methods,
but other obscure methods are sometimes encountered.
Rule-based shells are the most easily learned and their
knowledge bases are the simplest to understand.

Two methods of reaching conclusions are generally found
in shells:  forward chaining and backward chaining.
Forward chaining shells are best suited to data driven
problems.  Backward chaining shells are best suited to
diagnostic problems wherein a hypothesis is tested by
stepping backward through the rules.   It is important to
note that either search method can be implemented in the
other, although not very efficiently.  Some packages are
                        A-3

-------
                                    OSVER Directive #9028.OOa

capable of both methods of search, either in turn or
simultaneously.

Finally, an important feature to consider is the overall
si.ze of the shell.  The number of rules that it can
manipulate in one knowledge base is a quick measure of
this, but it is sometimes a false measure.  For example,
a package that can work with 2000 rules sounds superior
to a package that can work with 1000 rules until the user
realizes that the first package has no alternate
conclusion or multiple premise capabilities.  The second
system can thus store the same amount of knowledge in a
much smaller knowledge base.  Another factor to consider
is the ability to link smaller knowledge bases together
in order to exceed the inherent limit of the shell.

A.2.2.2  Creating the Knowledge Base (Attachment A.I,
Section 5.2)

Access to the knowledge base as it is being created is
critical.  Many shells have an interactive editor built
in that provides syntax and variable scope checking as
the rules are created.  Some shells provide access only
through an external text processor.  The ability to store
input and output in test files can tremendously simplify
testing a system, especially when the user input is long
or tedious.

Debugging features also vary widely.  Error messages can
be either insightful or cryptic, depending on the
particular shell.  Identification of the location of the
error  (not just reporting that the error exists) and
backward and forward tracing are variably supported.  A
feature seldom found but of great value is the
identification of unreachable goal states and unusable
rules.

A.2.2.3  Processing the Knowledge Base  (Attachment A.I,
Section 5.3)

Once the knowledge base has been created and made
accessible to a larger audience, it will have to provide
explanations to that audience of how and why it reaches
particular conclusions.  The text displayed upon request
and the points at which such requests can be made are
important issues to be considered.  Users will also
probably want a "what if" or "undo" capability in order
to explore the consequences of changing decisions.
                        A-4

-------
                                    OSUER Directive *9028.00a

The performance of an expert system shell can be
evaluated in several ways.  A set of standard benchmarks
should be developed to test rule processing speed,
capacity, and relative development time.  Typical
benchmarks include the Tower of Hanoi problem and the
animal classification game.  It is difficult to compare
the performance of shells in any great detail, but these
or similar tests give a rough indication of relative
performance.

Performance can also be gauged approximately by
determining whether the shell is compiled or interpreted.
Each has its advantages and disadvantages; the particular
needs of the user will determine which is better for a
given problem.  The shell's documentation may also give a
measure of performance, possibly in the form of logical
inferences per second (LIPS).

A.2.2.4  Portability (Attachment A.I, Section 5.4)

Portability of expert systems can be addressed from
several perspectives. -The value and usefulness of
knowledge bases is greatly enhanced if they are
represented in an easily comprehensible manner.  If they
are stored in a generic form (e.g., an ASCII file), .then
they can be ported across wprd processing packages and
displayed in a comprehensible form.  If the rule
structure is sufficiently English-like (as opposed to
complex computer -code), it can readily be interpreted by
a non-programmer expert.  While the knowledge bases
produced by most shells cannot be directly ported to
other shells, the ease with which they can be converted
to other environments should be considered.  In
particular, shells written in the same language may be
more compatible than shells written in different
languages.

A.2.2.5  User Interfaces (Attachment A.I, Section 5.5)

How the development and delivery systems interact with
the developer and user are significant issues.  The use
of pull-down menus, the availability of on-line help, and
a transparent operating system are among the many
desirable features that facilitate interfacing with the
system.  Graphic representation of the decision process,
either in static or dynamic form, can tremendously
enhance the ability of the system to present information
to the user.
                        A-5

-------
                                            OSUER Directive #9028.OOa

        A.2.2.6  Documentation (Attachment A.I, Section 5.6)

        As with any other type of software, good documentation of
        shells is critical.  The quality of the tables of
        contents, indices, illustrations, and examples in the
        developer's and user's references will drastically affect
        the ability of the developer and user to learn the shell
        and solve the problems that inevitably arise.  On-line or
        hard copy tutorials are useful features that can
        significantly reduce training time.

        A.2.2.7  General. Explanations, and Additional Features.
         (Attachment A.I, Sections 5.7 - 5.9)

        Because of the wide variety of features offered by
        different shells, no fill-in-the-blanks form can
        completely describe the attributes of all shells.
        Therefore, the evaluator should make general comments
        about the suitability of a given shell.  Attachment A.I,
        Expert System Shell Evaluation Form, provides ample room
        for explanations of other features or descriptions of
        other problems.  This is sometimes the most important •
        part of an evaluation; the evaluators should be liberal
        in their note taking.

        A.2.3  Overall Evaluation and Comments  (Attachment A.I,
        Sections 6 and 7)

        Finally, a conclusion has to be drawn about the overall
quality of the shell and its suitability to the problem at hand.
This is ultimately a subjective opinion since the evaluators may
have certain biases.  Therefore, if one evaluator is going to
review each of several packages being considered for a particular
application,  the same individual should evaluate them all in
order to permit a reasonably fair comparison.  If time permits
several evaluators to review each shell, then a higher degree of
objectivity can be attained.
                                                   f

A. 3     CONCLUSION

        The evaluation of a rule-based shell can be divided into
a review of the overall structure; the creation, processing, and
portability of the knowledge base; user interfaces;
documentation; and other miscellaneous information.  This
information should be recorded on a standard form, such as
provided in Attachment A.I, in order to facilitate comparisons of
different rule-based shells.  However, the evaluator must be
careful not to get caught up in the necessity of filling in every


                                A-6

-------
                                            OSUER Directive 09028.OOa

single blank on the form and thus lose sight of the larger
objective of getting a feel for the suitability of the shell to a
given problem.  Ultimately the question to be answered is, "Does
the package do the job as intended?"  Such fundamental questions
as "Does the package work as explained in the manual?" and "Does
the package have unexplained features?" are frequently overlooked
because they are difficult to quantify.

        Many commercial expert system shells were developed to
solve one specific type of problem and have since been modified
into general problem solvers.  The features added to make the
shell a general purpose tool are sometimes very obviously add-
ons; the underlying functionality is not enhanced by them.  The
evaluator must seek out the underlying functionality and not be
misled by the peripheral features.  Remaining objective in this
manner is essential to a proper evaluation.  Attachment A.2,
Shell Characteristics vs. Application Types, correlates the shell
features discussed here to typical expert system application
types as discussed in Chapter 1.
                                A-7

-------
                                                        OSWER DIRECTIVE *9028.00a
                          Attachment A.1
                       EVALUATION FORM
          FOR RULE-BASED  EXPERT  SYSTEM SHELLS
1. TESTER
   Name:
   Office:
              Date:
            Phone:
2.PRODUCT
   Name:    	
   Release/Version:  	
   Number of Installations:
   Run Time Version Price:
              Price: 	
              GSA (Y/N)?

              User Groups:
3. VENDOR
   Name:
   Address:  _
   Years in Business:
   Support Available:

4. REQUIREMENTS
   Minimal
   Hardware:   	
Financial Status:
Training Available:
   Recommended
   Hardware:
   Operating System:
   Systems it will run on:
   System tested on:

-------
Evaluation Form for Rule-based Expert System Shells                           OSWER DIRECTIVE #902a.00a
Page 2

5.  FEATURES
    If a feature is present, place an X in the box next to it If an explanation is necessary, place
a number in the box and add an explanation in the EXPLANATIONS section. If the feature is
not present and no explanation is needed, leave the box unmarked. An asterisk indicates a core
feature.

5.1   STRUCTURE
          Types of parameter values
*            Logical	L-l
*            Numeric	'—'
*            Enumerated	'—'
             Floating point	:	I—»
          Complex rule structures
             Premise (IF) section
                 Multiple  ANDs....:	L-1
               MultipleORs	 n
              Conclusion (THEN) section
                 Multiple  ANDs	LJ
          Confidence factor (deal with uncertainty)
              Bayesian probability	I—I
              MYCIN formula (CF1+CF2/100*(100-CF1))...	HH
              Other (specify)	,	'—I
          Multiple rule contexts	 '—'
          Rules refer-to parameters in direct contexts	I—•
          Math capability to combine or evaluate rule contexts	•—«
          Linkage to higher level programming languages (HLPL)
             Use HLPL to create values or get data	LJ
             Use HLPL to combine parameter values	I—J
             Use HLPL to perform mathematical operations	I—J

-------
_.....„,.._„     ., „                          OSWER DIRECTIVE #9023.003
Evaluation Form for Rule-based Expert System Shells
Page3
          Variety of knowledge representation forms
*            Rules [[[ L- 1
             Frames [[[ «— '
             Examples ...................... . .............................................. I— I
             Other (specify) [[[ I— 1
             Chaining mechanism
                 Forward  only [[[ I— I
              Backward only [[[ I — I
*         Capacity (number of rules)
             Object-oriented structure ............ . ...................................... ' — >
             Extensible expert system shell (explain) .............. •. .................. ' — I

5.2    CREATING THE KNOWLEDGE BASE
          Interactive rule-building with  error checking .................... . ............ I— I
*         Test file processing
             Save an interactive session and replay it .......................... . ...... L— I
             Run a test file [[[ I — •
*         Debugging features
             Meaningful error messages.... ................... . ........................ > — I
             Help in locating the source of the problem. .............................. <— I
             Interaction between run, debug, and edit ................................ L— '
             Check for  unused  rules ........................................ '. ............ I— '

-------
Evaluation Form for Ride-based Expert System Shells                        OSWER DWECTWE #9028.00*
Page 4

          Undo capability	*—'
             Truth Maintenance	*—*
        •  Performance
             Compiled	'—'
             Interpreted	•—'
             Good response time (explain)	'—•

5.4   PORTABILITY
       Knowledge base stored in ASCII file (or other standard)	'—«
 5.5   USER INTERFACE
 *     Invisible operating system	.'	'—'
       Menus	 '—'
       Commands	:	I—'
       On-line   help	'—'
 *     Logical, intuitive operation	«—•>
 *     Protects against user errors	'.	I—'
 *     Provides meaningful error message	I—I
       Graphics display capability
          Show decision structure	I—I
          Show data structure	,	'—I
          Illustrate parameter or derived values	I—'
          Show relations between parameters	'—I
          Other	CH
       Mouse, icons,  pop-up menus,  windows	I—I

 5.6   DOCUMENTATION
       Easy-to-use table of contents	'—I
 *     Well organized	 '—'
 *     Well  written	:	

-------
Evaluation Form for Rule-based Expert System Shells
Page 5
      Good  examples [[[ I— I
      Illustrated.... [[[ L— 1
*     Good index ........... . [[[ HU
      Programmer's  reference [[[ I— J

-------
Evaluation Form for Ride-based Expert System Shells
Page 6

6.  OVERALL EVALUATION
    Does this package meet your expectations              '—'          •—'
,    for a product of its type?                            Yes           No

    Does this package meet all the core requirements        '—'          '—'
    in the evaluation form?                             Yes          No

    Do you feel that this package should be                I—'          '—'
    listed as a preferred package                        Yes          No

    How many hours did it take you to become
    reasonably proficient in the package's use?                  	hours

    The following items are rated, on a scale of 1 to 10. A poor rating is indicated by 1 and an
 excellent rating is indicated by 10.

 Performance   	        Documentation   	      Recommendation    	
 Ease of Use   	        Ease of Learning  	      Flexibility   	

 7. COMMENTS

-------
Shell Characteristics vs. Application Types
r Probably Required
O Possibly Required
Q Not Likely to be Required
Diagnosis &
Classification
to
'co
"J3 S
< e
nl —
Q ofl
C/J
1 c <8
'co c
CD >,
Q CO
Prediction &
Simulation
Monitoring
0
CO
1
0)
to
Intelligent
Assistant
Planning &
Scheduling
2
c
o
O
Inference Approach
Backward Chaining
Forward Chaining & Reasoning
BC, FC, & Forward Reasoning
Hypothetical Reasoning
Object Oriented
Blackboard
Induction
•
0
•
•
0
•
•
O
•
•
•
0
•
0
o
•
•
•
o
•
o
Q
•
O
O
•
o
a
O
•
O
o
o
o
Q
0
• •
0
O
o
o
o
0
0
•
O
0
o
o
O
•
•
•
O
•
Q
0
•
O
Q
•
Q
O
Object Description
Frames
Frames with Inheritance
Objects
Parameter Values Pairs
Logic
Rules
Certainties
•
•
o
o
o
o
•
o
0
o
o
0
o
•
•
•
o.
o
o
o
o
o-
o
•
Q
O
0
o
O
0
o
o
o
o
0
o
o
o
o
o
0
o
o
o
0
Q
O
o
o
•
•
0
Q
O
O
o
O
O
•
0
o
o
o
Actions
Rules
Examples
Logic
Messages
Procedures
•
•
•
o
0
•
0
•
o
o
•
o
•
•
•
•
o
•
•
•
•
Q
0
o
•
•
o
o
o
0
•
0
o
0
o
•
Q
O
•
•
•
0
o
•
•
                                                                            to

                                                                            8
                                                                            CD

-------
                                            OSUER Directive #9028.OOa

        APPENDIX B

        GLOSSARY


AZ.  See artificial intelligence.

Algorithm.  A formal procedure guaranteed to produce correct or
optimal solutions.

Architecture.   (1) The organizing framework imposed upon
knowledge applications and problem-solving activities.   (2) The
knowledge-engineering principles that govern selection of
appropriate frameworks for specific expert systems.

Artificial intelligence.  The subfield of computer science
concerned with  developing intelligent computer programs.  This
includes programs that can solve problems, learn from experience,
understand language, interpret visual scenes, and, in general,
behave in a way that would be considered intelligent if observed
in a human.

Backtracking.   A search procedure that makes guesses at various
points during problem-solving and returning to a previous point
to make another choice- when a guess leads to an unacceptable
result.                     .                  •

Backward chaining.  An inference procedure or strategy .used by
the inference engine where the system starts with what it wants
to prove  (e.g., the goal), and tries to establish the facts it
needs to reach  that goal.  The text of the goal is matched
against the conclusion (i.e., in the example "if the sky is
cloudy, then the forecast might include rain" is a conclusion)
for each rule to determine whether that rule will contribute
information to  the resolution of the goal.  If the conclusion of
a rule matches  the goal, then the premises of the rule are
considered in turn.  Each of the premises is considered to be an
intermediate goal.  Results of evaluating each goal are  stored in
memory to be used when evaluating subsequent goals.

Baseline.  The  set of life cycle products and other related
documentation which officially comprises the system at a given
point in time.

Blackboard.  A  database globally accessible to independent
knowledge sources and used by the:  to communicate with one
another.  The information they provide each other consists
primarily of intermediate results of problem solving.
                                    B-l

-------
                                            OSUER Directive #9028.OOa

Chaining, backward.  See backward chaining.

Chaining, forward.  See forward chaining.

Conflict resolution.  The technique of resolving thi problem of
multiple matches in a rule-based system.  When more than one
rule's antecedent matches the database, a conflict arises since
(1) every matched rule could appropriately be executed next, and
(2) only one rule can actually be executed next.  A common
conflict resolution method is priority ordering, where each rule
has an assigned priority and the highest priority rule that
currently matches the database is executed next.

Control structure.  Any procedure, explicit or  implicit, that
determines the overall order of problem-solving activities; the
temporal organization of subprocesses.

Domain expert.  A person who, through years of  training and
experience, has become extremely proficient at  problem solving in
a particular domain.

Domain knowledge.  Knowledge about the problem  domain, e.g.,
knowledge about geology in an expert system for finding mineral
deposits.

End user.  The person who uses the finished expert system; the
person for whom the system was developed.

ES.  See expert system.

Expert system.  A computer program- that uses expert knowledge to
attain high levels of performance in a narrow problem area.
These programs typically represent knowledge symbolically,
examine and explain their reasoning processes,  and address
problem areas that require years of special training  and
education for humans to master.

Expertise.  The set of capabilities that underlie the high
performance of human experts, including extensive domain
knowledge, heuristic rules that simplify and improve  approaches
to problem-solving, metaknowledge and metacognition,  and complied
forms of behavior that afford great economy in  skilled
performance.

Expert system development environment.  The programming language
and support package used to build the expert system.

Explanation.  Motivating, justifying, or rationalizing an action
by presenting antecedent considerations such as goals, laws, or


                                    B-2

-------
                                            OSUER Directive #9028.OOa

heuristic rules that affected or determined the desirability  of
the action.

Explanation facility.  That part of an expert system that
explains how solutions were reached and justifies the steps used
to reach them.

Fact.  A proposition or datum whose validity is accepted.

Forward chaining.  One of several control strategies that
regulate the order in which inferences are drawn.  In a rule-
based system, forward chaining begins by asserting all of the
rules whose IF clauses are true.  It then checks to determine
what additional rule might be true, given the facts it has
already established.  This process is repeated until the  program
reaches a goal or runs out of new possibilities.

Frame.  A knowledge representation method that associates
features with nodes representing concepts or objects.  The
features are described in terms of attributes (called slots)  and
their values.  The nodes form a network connected by relations
and organized into a hierarchy.  Each node's slots can be filled
with values to help describe the concept that the node
represents.  The process of adding or removing values from  the
slots can activate procedures (self-contained pieces of code)
attached to the slots.  These procedures may then modify  values
in other slots, continuing the process until the desired  goal... is
achieved.  Also called Data Directed Reasoning.

Heuristic.  A rule of thumb or simplification that limits the
search for solutions in domains that are difficult and poorly
understood.

Inference engine.  That part of a knowledge-based system  or
expert system that contains the specific procedures and
algorithms for using the rules/heuristic in the knowledge base to
solve a problem.  The inference engine processes the domain
knowledge (located in the knowledge base) to reach'new
conclusions.

Knowledge.  The facts, beliefs, and heuristic rules a computer
program must have to behave intelligently.

Knowledge acquisition.  The process of extracting, structuring,
and organizing knowledge from some source, usually human  experts,
so it can be used in a program.

Knowledge base.  The portion of a knowledge-based system  or
expert system that contains the domain knowledge.


                                    B-3

-------
                                            OSWER Directive #9028.OOa


Knowledge-based system.  A program in which the domain knowledge
is explicit and separate from the program's other knowledge.

Knowledge engineer.  The person who designs and builds the expert
system.  This person is usually a computer scientist experienced
in applied artificial intelligence methods.
                      *
Knowledge engineering.  The discipline that addresses the task of
building expert systems; the tools and methods that support the
development of an expert system.

Knowledge management.  The process of formally controlling any
changes or additions to the knowledge base in order to maintain
expert system integrity.

Knowledge representation.  The process of structuring knowledge
about a problem in a way that makes the problem easier to solve.

Knowledge source.  Generally, a body of domain knowledge relevant
to a specific problem.  In particular, a codification made
applicable for an expert system.

Natural language.  The standard method of exchanging information
between people, such as English  (contrasted with artificial
languages, such as programming languages).

PROLOG.  An Artificial Intelligence programming language based on
predicate calculus.  PROLOG is short for the French words
Proqramination en Locricrue.

Real-world problem.  A complex, practical problem which has a
solution that is useful in some cost-effective way.

Representation.  The process of formulating or viewing a problem
so it will be easy to solve.

Robustness.  That quality of a problem solver that permits a
gradual degradation in performance when it is pushed to the
limits of its scope of expertise or is given error laden,
inconsistent, or incomplete data or rules.

Rule.  A formal way of specifying a recommendation, directive, or
strategy, expressed as IF premise THEN conclusion or IF condition
THEN action:

Rule-based methods.  Programming methods using IF-THEN rules to
perform forward or backward chaining.
                                    B-4

-------
                                            OSUER Directive *9028.00a

Rule-based program.  A computer program, that constitutes a  module
of heuristic knowledge.

Search.  The process of looking through the set of possible
solutions to a problem in order to find an acceptable solution.

Shell.  The common term for expert system development
environment.

Symbol.  A string of characters that stands for some  real-world
concept.

Symbolic reasoning.  Problems solving based on  the application of
strategies and heuristic to manipulate symbols  standing for
problem concepts.

Tools for knowledge engineering.  Programming systems that
simplify the work of building expert systems.   They include
languages, programs, and facilities that assist the knowledge
engineer.

Tree structure.  A way of organizing information as a connected
graph where each node can branch into other nodes deeper in the
structure.

User.  A person who uses an expert system, such as an end-user, a
domain expert, a knowledge engineer, a tool builder,  or a
clerical staff member.
                                    B-5

-------
                        OSWER DIRECTIVE #9028.003
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
    Part 3: Practice Paper
     Data Management
During the System Life Cycle
      January, 1989
                            J

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM LIFE CYCLE
                                TABLE OF CONTENTS
       t
1.  Introduction	     1
2.  Selecting A Data Management Approach  	 •  5
3.  An Overview Of Data Management Topics	'	    15
4.  Data Modeling Activities	    23
5.  Data Design Activities	    29
6.  Data Stewardship Related Activities	    32
7.  Data Documentation Activities	    40
8.  Terms And Reference Materials  	    44

                                    EXHIBITS
1-1      Practice Paper Overview                                              2
2-1      An Overview of Data Management During the System Life Cycle          6
2-2      Scoping Your Project's Impact                                        8
2-3      Selecting Your Data Management Approach                              9
2-4      Data Management Plan Outline                                        13
3-1      Data Management Topics Through the Life Cycle                       16
4-1      Entity List Example                                                 24
4-2      Conceptual Data Model Example                                       26
4-3      Logical Data Model Example                                          27
6-1      The Data Steward Role                                               34
6-2      Functional Data Roles                                               35
6-3      Determining Data Stewardship                                        39
7-1      Data Dictionaries in the System Life Cycle                          43

-------
         PRACTICE PAPER FOR DATA MANAGEMENT  DURING THE SYSTEM LIFE CYCLE
                                    CHAPTER 1

                                  INTRODUCTION
1.1 Practice Paper Purpose

    This  practice  paper  provides details to project managers concerning their
responsibilities  for  data  management1  under  the  Office  of Solid  Waste and
Emergency  Response (OSWER) System Life Cycle Management Guidance.   The practice
paper  describes  data  management  during  the  system life cycle,  and provides
guidance  concerning  major  topics  that  should be addressed by project teams.
Data  management  begins  during the Concept Phase,  proceeds as requirements are
defined  and software is implemented, and continues  until the application system
is terminated or replaced.

    This practice paper has three primary purposes:

    o  Focus  each  project  manager's  attention upon  ensuring  that the data
       provided by OSWER systems meets program requirements;
                                         •
    o  Facilitate successful data management for each project;

  .  o  Provide a common approach to data management  across OSWER.

    This  practice  paper constitutes a section of Part 3 of the Office of Solid
Waste and Emergency Response (OSWER) System Life Cycle Management Guidance.

1.2 Practice Paper Topics

    Exhibit  1-1  provides  an overview of the structure of this practice paper.
Topics addressed in this practice paper include:

    o    Selection of an approach to data management for a project;

    o    The  details of data management topics applicable to OSWER's Life Cycle
         Management Guidance;

    o    The  flow of major data management activities,, including data modeling,
         data design, data stewardship, and data documentation.
*0ata  management includes data-related activities such as logical data modeling
during  requirements definition, data base design, data base management, and the
documentation of data-related decisions and products.

                                        1                           May 31, 1988

-------
                                          EXHIBIT 1-1
                 DATA MANAGEMENT PRACTICE PAPER OVERVIEW
   OFFICE OF SOLID WASTE
 AND EMERGENCY RESPONSE
          (OSWER)
      PRACTICE PAPER
            FOR
    DATA MANAGEMENT
           DURING
  THE SYSTEM LIFE CYCLE
          Prepared for
Office of Program Management and Technology
     Information Management Siajf
                                                                                         CIIAITER «
                                                                                           TEHMS
                                                                                           AND
                                                                                         REfKkKNCE
                                                                                         MATERIALS
                                                                                    CHAPTER 7
                                                                                DATA DOCUMENTATION
                                                                                    ACTIVITIES
                  CHAPTER t
               DATA  STEWARDSHIP
               RELATED ACTIVITIES
                                                                          CHAPTER 5
                                                                         DATA DESIGN
                                                                          ACTIVITIES
                                                                     CHAPTER 4
                                                                   DATA MODELING
                                                                     ACTIVITIES
                                                          CHAPTER 2
                                                          SELECTING
                                                       A DATA MANAGEMENT
                                                          APPROACH
                                                      CHAPTER I
                                                    INTRODUCTION
   CHAPTER 3
 AN OVERVIEW OF
DATA MANAGEMENT
    TOPICS

-------
                                                                OSWER DIRECTIVE #90SB.OCIa

         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


1.3 Data Management Responsibilities

    Project  managers  are responsible for implementing the guidance within this
practice  paper  when  OSWER  systems  and  data  bases are built.  Each project
manager  will  select  an  appropriate  approach  to data management for her/his
project  and  ensure that this approach is documented in a Data Management Plan.
Project managers will use the Project Management Plan to schedule the activities
detailed in the Data Management Plan.

    Project  managers  are  also  responsible  for  ensuring that a programmatic
requirement  for  data  is  identified  and documented before data is collected,
processed,  stored,  and  distributed.    They  must work closely with the OSWER
program  offices  affected by a project to ensure that this basic responsibility
is met.

    OSWER  program  offices  are  responsible for supporting the data management
process  of  each  project.    This  responsibility includes providing competent
programmatic  personnel to identify data requirements and to define the meaning,
allowable  values,  edit  criteria, and the- level of quality and security of the
data.   Program offices that assume data stewardship responsibility (see Chapter
6)  also determine who will collect data and who will ensure its integrity after
it is collected.

1.4 Basic Principles

    The data management activities performed during the systems development life
cycle are based upon the following basic principles:

    o  Data is a valuable resource.  Data is collected, stored, and used to
       support  critical OSWER program activities and decisions, making-accurate
       and timely data an important OSWER resource.

    o  Data  is defined separately from the technology used to collect and store
       it.    OSWER  data  requirements  are recorded clearly prior to designing
       automated  data collection and storage methods, so that program needs are
       understood and recorded.     i

    o  Accurate  information  about .data is essential.  Effective management of
       data  collected  by  OSWER  requires that accurate information about data
       (metadata) be kept.

    o  Common data management guidelines, methods, and tools are used.  A common
       approach  to  defining,  modeling,  designing,  and documenting data will
       improve  OSWER's  data  quality  and  make  it easier to share data among
       systems and offices.

1.5 Why Focus Upon Data Management?

    Automated and manual systems provide information to the OSWER program.  This
information  is used, for example, to make decisions affecting public health and
safety,  environmental  quality,  and  the  use  of  public funds.  Without this
information,  the Office of Solid Waste and Emergency Response could not perfora
its  mission.   The data collected, created, stored, processed, and disseminate"
by  OSWER  systems  are  used  to create the information OSWER needs to operate.

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING  THE  SYSTEM LIFE  CYCLE


Since  Information  must be based upon accurate data, future OSWER-funded  system
development  projects  will focus upon the provision of accurate  and  timely  data
to OSWER programs.

    Benefits of increasing the focus upon data management include:

    o  Ensuring  that  data  collected and disseminated on behalf of  OSWER meets
       programmatic  requirements fully, including  requirements for accuracy and
       timeliness;

    o  Improving  management  decision-making by providing better access  to  more
       accurate and timely data;

    o  Increasing  productivity  in  the  information   collection and processing
       activities  of  OSWER  offices  as the understanding and use of available
       data increases;

    o  Ensuring  that  existing  data  can  be shared  to the maximum  practicable
       extent, avoiding the cost of redundant data collection and storage;

    o  Reducing  the  cost  of  system maintenance and the time needed to modify
       implemented systems by designing more stable and flexible data bases.

1.6 How to Use this Practice Paper

    Before  you  read  this  practice  paper,  read  the OSWER System Life Cycle
Management   Guidance,  including  the  Data  Management  Plan  exhibits  within
different  sections  of  the  guidance.  Then, read this practice paper to learn
more information about the topics covered during the system  life cycle.

    Use  this  practice  paper  to  guide  the  development  of a data management
approach  that is  appropriate for your project.  Then, document your approach  in
the  project's  Data Management Plan as the project evolves.  Add information  to
the plan, and modify existing information to reflect the current approach at any
point  in time.

1.7 Project Participation  and Coordination Practice Paper

    Read  the  Project  Participation  and  Coordination  Practice  Paper before
beginning  to  document  your  data  management approach.  The participation and'
coordination   practice    paper  details  who  should  be   involved  in  project
activities,  including  the  activities  you  will define in the Data Management
Plan.

1.8 Configuration  Management Practice Paper

    This   practice   paper   describes  change  control  of the  baselines   of
requirements,  specifications,  and operating  functionality  which are documented
in  the  products  of the system life cycle.  For instance, the Requirements Data
Dictionary  is a product which  is initially created during the Definition Stage.
Changes  to  this  product  are controlled using the procedures described in the
Configuration  Management  Practice  Paper.    Read the Configuration Management
Practice Paper before preparing your Data Management Plan.

-------
                                                                 uJWER DIRECTIVE #9028.00a

          PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM  LIFE CYCLE
                                     CHAPTER 2

                       SELECTING A DATA MANAGEMENT APPROACH
 2.1  Introduction
     This  chapter  provides guidance  for the  project manager who is preparing  an
 approach to managing data-related  activities, products, and decisions during the
 systems  life cycle.  Your data management  approach will have a major impact upon
 the success of your project and the information provided to OSWER programs.

 2.2 What Is a Data Management  Approach?

     The   data-related  activities, products,  and decisions you decide to address
 during   the  system  life  cycle constitute your data management approach.  Your
 approach  also  includes   the   degree of  r.igor you follow when performing these
 activities  and  the level  of  formality you choose when documenting data-related
 life cycle products and decisions.

     Exhibit  2-1  provides   an overview of data management activities during the
.life cycle.    Chapter -3   provides  detailed  information  about  data-related
 activities and products on  a stage-by-stage basis.

 2.3 Why  Choose A Data Management Approach

     Selecting and implementing a data management approach that is appropriate  to
 your project is'a key to  your  project's success.  If you choose an approach that
 doesn't   address data dictionary issues as part of a large, high impact project,
 you  will   increase  the   risk  of  time and  cost overruns for your project, and
 likely   will  increase maintenance costs  for the completed system and its data.
 This chapter will give you guidance  in determining how much rigor and formalism
 are required for your project.

     One   criterion  which  stands but as a major factor in determining your data
 management approach is the  degree of  data  sharing.  Data sharing includes use  of
 one  information  system's   data  by  a second system, and using the same data  by
 multiple  functions  using   a   single information  system.   If data sharing-  is
 extensive,  you  should  choose a rigorous and formal data management approach.
 Following  this type of approach will minimize unexpected, negative impacts upon
 your system and the OSWER programs it will support.

 2.4 Determining Your Data Management  Approach

     Adjust  your  project's data  management approach  to fit the scope of the
 system  'you  develop.    The degree of data sharing, the scope of organizational
 impact,  and cost of the planned system should be reviewed to determine your data
 management  approach.    As the relative degree of each criterion increases, the
 level of rigor and formalism you select for your data management approach should
 .increase accordingly.
                                                                    May  31,  1988

-------
                               EXHIBIT 2-1
     AN OVERVIEW OF DATA MANAGEMENT ACTIVITIES
                 DURING THE SYSTEM LIFE CYCLE
                                     INITIATION
                                      Identify
                                     Information
                                      Need
                                                        Define
                                                       High Level
                                                        Data
                                                      Requirements
                          Archive
                          Data&
                          Son ware
EVALUATION
                                                            DEFINITION
        Evaluate
        Data Base
                                                             Define
                                                            Detailed
                                                             Data
                                                           Requirements
                                    DATA
                                MANAGEMENT
                                 DURING THE
                             SYSTEM LIFE CYCLE
PRODUCTION
           Operate
          and Maintain
          Data Base
                                                           Design
                                                           Physical
                                                           Data Base
                                                       Create
                                                      and Test
                                                      Data Base
                 Convert
                Production
                  Data
                                      Establish
                                      and Test
                                      Production
                                      Data Base
                                                    DEVELOPMENT
                         IMPLEMENTATION

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM  LIFE CYCLE


2.4.1    Scoping The Project

         Begin  by  determining  the  scope  of data sharing,  the organizational
impact, and the cost of the project.  Extensive data sharing often increases the
impact  of  a  system  upon the agency.  Cost is not as important a  criterion  in
scoping  your  project, but a large investment in a system should be accompanied
by  a  formal  approach  to  planning and designing the data used to support the
system.

         Refer  to  Exhibit  2-2, a chart that will help you scope your project.
If  any  of  the statements listed under the high impact column (column"two) are
true for the project, then your project should be considered high  impact.

         If  your  project is not high impact, evaluate each criterion in  column
one  against the information in the medium impact column (column three).   If any
of  the statements under column three apply to your project, then it is a  medium
impact  system.    If your project is neither high or medium impact, it is a low
impact system for the purpose of determining a data management approach.

2.4.2    Select A Preliminary Data Management Approach

         Next,  select  the  data management approach (preliminary)  you will use
for  the  project.    Refer  to  Exhibit  2-3  to help select  a preliminary data
management  approach.  Choose the decisions, products, and activities you  should
include  in your data management approach, and determine'the degree  of formalism
you  will  use  to document decisions and products.  Keep in mind that a primary
objective  of every system project is to provide accurate and  consistent data  to
the system users.           •

High Impact Projects

         Follow  the  guidance provided in Chapters 3 through  6 of this practice
paper  if you have a high impact project.  Conceptual data modeling,  logical data
modeling,  and  physical  data  base  design are performed.  Document the plans,
activities, decisions, and products called for in these chapters formally.  Data
documentation  is  stored in automated requirements, design, and production data
dictionaries.    If  you are managing a high impact project, appoint a full-time
person to coordinate data management activities for the project on your behalf.

Medium Impact Projects

         Medium  impact  projects  require  subjective  judgments concerning the
components  you select for your data management approach.  While conceptual data
modeling,  logical  data  modeling,  and physical data base design should all  be
performed,  the level of formalism you select for documenting  your decisions and
products  may  be  less.    If you are managing a medium impact project, you may
complete  data  dictionary  documentation  differently  than  for  a high impact
project.  Although requirements, design, and production data dictionaries should
be kept, only the design and production dictionaries need to be automated.

-------
                                              EXHIBIT 2-2
                         SCOPING  YOUR  PROJECT'S  IMPACT
        CRITERION
       HIGH IMPACT
    MEDIUM IMPACT
    LOW IMPACT
  Data Sharing
Dau are created, collected, and/or
used to support operations of
multiple OSWER program offices,
legions! offices, and/or state agencies
            and
Data are  used by multiple EPA
offices to provide environmental
information to OSWER management,
the public, and Congress
 Data are created, collected, and/or
used by multiple OSWER program
offices. Regional offices, and/or
state agencies
 >ata are created, collected
and/or used within a single
OSWER program office;
 o dau are created by or
 ollected from Regional
 iffices, and/or state
 igencies
   Organizational Impact
  Deemed a national priority system.1
  and is placed on the President's list
  of national priority systems
 Requires hit support from
 multiple OSWER program
 offices, or from Regional
 offices, and/or from state agencies
 Requires FTE support from
 a single OSWER program
 office; no FTE support
 required from other
 OSWER offices. Regional
 offices,-or from state
 agencies
   Cost
                                 Cost exceeds the available budget
                                 support of the sponsoring
                                 OSWER office, or the total life
                                 cycle cost meets OMB reporting
                                 requirements
                              Sponsoring OSWER office
                              can provide sufficient
                              funding with available
                              budget support, and all
                              costs are below OMB
                              reporting requirements
1 A national priority system is a system deemed to be critical to the government's mission.
Please note: If any criterion under a column is met, then the project's application is treated as the type of project described in that column.
                                                           8

-------
                                   EXHIBIT 2-3
                SELECTING YOUR DATA MANAGEMENT APPROACH
PROJECT SCOPE
        DATA
    MANAGEMENT
     ACTIVITIES
  DOCUMENTATION
         OF
DECISIONS/PRODUCTS
      DATA
   DICTIONARY
DOCUMENTATION
HIGH IMPACT
    RIGOROUS
   EXECUTION
(See Chapters 3, 4,5,6)
MEDIUM IMPACT
    MODIFIED
LOW IMPACT
    MODIFIED
     FORMAL
   (See Chapter 3)
     MODIFIED
    MODIFIED
   FORMAL  -
(See Chapters 3,7)
Requirement, Design, and
Production data dictionary
in automated form.
   FORMAL
Requirement, Design, and
Production data dictionary
in automated form.
   FORMAL

Production data dictionary

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCtE


Low Impact Projects

         Low  impact  system  projects  require  a  less  rigorous  approach  to
activities than the other two classes of projects.  Logical data modeling should
be  done  with  the  resulting  data  requirements documented.   Documentation of
production  data  bases  should  also  be  produced and stored  manually or in an
automated  data  dictionary.  Ensure that the activities and products you select
for your approach are appropriate to the scope of your project.

2.4.3    Consult With The OSWER Data Administrator

         After  selecting  a  preliminary  data management approach, contact the
OSWER  data administrator in the Office of Policy, Management and Technology for
assistance.    Review  the  project's scope and your preliminary data management
approach  with  the  data  administrator.   The data administrator will help you
adjust your data management approach to fit the scope of your project and advise
you of support capabilities OSWER can provide for your project.

2.4.4  Finalize and Document Data Management Approach

         Finalize your data management approach and document it in the project's
data  management  plan.    Exhibit  2-4  .at the end of this chapter provides the
outline  of  topics  for  your  data  management  plan.   This  plan will provide
valuable  input  into  project  planning  and staffing activities, including the
preparation of the Project Management Plan.

2.5 Prepare the Data Management Plan

    OSWER  intends  to  increase  the  degree of data integration and sharing to
provide  improved information management for mission support.  Implementation of
increased  data  integration and sharing will be supported by automated software
tools  for  system  and  data  base  development, and an OSWER-wide inventory or
directory  of  the  data  OSWER  has collected.  Plan to take advantage of these
capabilities when you prepare your data management plan.

    Record  the  data  management approach in your data management plan early in
the  Concept  Phase.    Include  those  topics  described in Chapter 3 which are
appropriate.    Then, refer to the plan at the beginning of each stage, revising
it  as  you  find necessary.  See Exhibit 3-1 for a summary of  this creation and
revision process.

    As  a project manager, you will find the details in the data management plan
helpful  when  you select staff for your project.  Before you prepare your plan,
you  should  read  Chapters 4 through 8 of this practice paper  to understand the
work you .will be planning.

    Pay particular attention to coordinating activities in four major areas when
preparing the substance of your data management plan.

2.5.1  Data Stewardship

       Before you can be sure you have identified the relevant  data requirements
for  a  project,  you  must  first  determine  who is authorized to define these
requirements.      Data  stewardship  assigns  the  functional   data  roles  andj

                                       10

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


responsibilities to the organizations which exercise control  over data on behalf
of OSWER.

       Identify  the data steward, definer, and primary user for each high-level
data  requirement  identified  during zhe Concept Phase.  Consult with the OSWER
data administrator and read Chapter 6 before planning activities related to data
stewardship.

2.5.2  Data Modeling

       Data modeling activities include identifying  data requirements, defining
the  meaning  of  the  requirements,  and structuring the data requirements into
logical  models.   These logical models represent the mission-oriented structure
of  the  data.   Data modeling and data models are completely independent of all
technology used to store and access data.

       When planning data modeling activities, assign an individual with systems
analysis  skills  and  OSWER program knowledge to lead the effort.  Be sure that
"users" of the system and data assume the data definer role described in Chapter
6.    It  is  the  user  perspective  which  is  critical to the success of data
modeling.

       You begin data modeling in the Concept Phase and finish the first version
•of  the  data  model in the Definition Stage.  The data model is modified at any
point  in  the  life  cycle  when  data' requirements change.  Since a number of
requirements  changes  will  normally  occur  during  the Design and Development
stages,  you  should  also  anticipate changes to your logical data model during
these  stages.    See  Chapter  4  ,of this document for more information on data
modeling.

2.5.3  Physical Data Base Design

       Physical data base activities build upon the results of the data modeling
activities to produce a physical data base design.  This data base design should
be done before creating data definition language statements that will be used to
create  the physical structure of data bases.  Data design activities should not
begin  until  after  the first version of the data model is complete, since data
design activities build upon the results of the data model.
                                                              /
       The  individual  assigned  responsibility  for data base design must have
technical  expertise in the data base management system or file structures being
used.    You  should  schedule  walkthroughs of the design, just as you schedule
walkthroughs  of  your  software program designs.  The physical data base design
walkthroughs  should  be attended by the data designer, programmers who will use
the  data  structures, the lead data modeling person, and the person responsible
for data management.

       An  initial  version of the data base design will normally be required by
the  end  of  Design Stage, so that physical data structures (data bases) can be
created to support programming and unit testing during Development.  See Chapter
5 for more information on data base design activities.
                                        11

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM'LIFE CYCLE


2.5.4  Data Documentation

       Collecting,  storing,  and using data about data (metadata) 1s crucial  to
the  success  of  system development project activities.   As the Individual  with
primary data management responsibility for your project,  you should plan for and
manage metadata effectively.  Define your data documentation activities early in
the Concept Phase, and record them in the Data Management Plan.

       You should record metadata describing the meaning, attributes, properties
and  structures  of  the data in the logical data model and data base design,  as
well  as  data  stewardship  information.   Normally, metadata will be stored  in
automated  software  systems called data dictionaries.  You must plan to use the
metadata of prior stages in subsequent stages whether automated  software 1s  used
or not. See Chapter 7 for more information about data documentation activities.
                                        12

-------
                             EXHIBIT 2-4
           DATA MANAGEMENT PLAN  OUTLINE
     This   topical  outline contains all the topics that Tiay be  contained  in the
Data  Management Plan for a High  Impact Project.  Taken together,  the  topics you
enter  into your  Data  Management Plan document your project's data  management
approach.

o    Information Need
          Document the Information Need
          Missions Supported
          Scope of the Need

o    Data  Steward Organizations
          Lead Organization Responsibilities
          Other Organizations Roles
          Data Definers For The Project

o    Concept Phase
     —  Entity List
     —  Entity Definitions
     —  Entity Identifiers
     —  Conceptual Data Model '
     —  Likely Sources of Data
     —  Information Flow/Data Model  Validation
     —  Data Distribution Plan
     —  Information Collection Burden            .

o    Definition Stage            .      '
     —  Interview Plans
     —  Data Analysis By Process
     —  Entity Normalization
     —  Conceptual Data Model Revision
     —  High-Level Data Requirements Revision
     —  Logical Data Model
     —  Requirements Data Dictionary
     —  Data Flow/Logical Data Model Validation

o    Design Stage
     —  Logical Data Model Revision
     —  Physical Data Base Design
     —  Design Data Dictionary

o    Development Stage
     —  Data Structures for Programming  Support
     —  Data (structure) Revision Approach
     —  Data Backup, Logging and Recovery Plans

o    Implementation Stage
     —  Testing Support (See Testing Support  Plan)
     —  Cutover Plans
     —  Production Data Dictionary
                                        13

-------
                           EXHIBIT 2-4
DATA  MANAGEMENT PLAN OUTLINE  (Continued)
o    Production Stage
     —  Data Base and Metadata Management
     —  Support of Configuration Management
     —  Backup, Recovery and Restart
     —  Role of the Custodian

o    Evaluation Stage
     —  Audit and Evaluation Support Plan
     —  Response to Evaluation Report

o    Archive Stage
     —  Data Base DDL and Metadata Disposition
     —  Data Disposition
     —  Cutover Procedures

o    Data  Documentation Responsibilities
          Creating Data Documentation
          Maintaining Existing Data Documentation

o    Data  Quality Assurance Plans
          Responsible Organization
          Milestones and Staffing
          Data Quality Objective Monitoring Plan

o    Data  Security Requirements and Strategy
          Sensitive Data

o    Data  Life Cycle Methodologies and Tools
          Metadata Management Approach
          Development & Installation Phase
          Data Management Software
          Operations Phase

o    Data  Conversion Strategy
                   •

o    Data  Conversion Plan
          Sources
     —   Media
          Load Programs Required
          Schedule and Staffing
          Validation

o    Plan  For Physical Flow Of Data

o    Data  Testing S.rategy

o    Testing Support
          Kinds of Test Data Bases Required
          Test Data Provision
          Performance Validation Plan
          Responsible Organization
          Projected Testing Support Needed
                                     14

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM  LIFE CYCLE
                                    CHAPTER 3

                      AN OVERVIEW OF DATA MANAGEMENT TOPICS
3.0 Introduction
    This  chapter  provides additional details of the topics that you  can  select
as  a part of your data management approach.  If you select one of these topics,
refer  to  this chapter to learn what should be included in your Data  Management
Plan.    Keep  in  mind  that  the data management approach and much of the Data
Management  Plan  are  prepared during the early part of Concept Phase.  In many
instances  you  may not want to enter all of the details of your data  management
approach in the plan; if not, at least indicate which topics you are addressing.

3.1 Initiation Phase Topic

3.1.1  Information Need

    Prepare  a  brief.summary of the information need created by the information
management  problem  discussed  in  the  Initiation  Decision  Paper.    If it is
necessary  to  collect new information or access existing information  to solve a
problem,  then you should select a data management approach and document it in a
Data  Management  Plan.    Record the programmatic missions with the information
need, and review this information at the start of each phase/stage to  be sure it
is still valid.

3.2 Concept Phase Top-ics

3.2.1  Data Steward Organizations
                                                              ;
    Document   the   data   stewardship  responsibilities  of  the  organization
sponsoring  the  project.    Then, record the data steward, data collector, data
definer,  and  primary  data  user  for  each  data entity identified  during the
Concept   and  Definition.  See  Chapter  6  for  more  information 'about  data
stewardship.

3.2.2    High Level Data Requirements

    -- Entity List
         - Data Entities  about which data  is needed (e.g., Employee)

    -- Entity Definitions
         - Programmatic definitions of each entity
2
 Data entity is a person, place, thing, concept, or event about which OSWER will
store data.

                                       15                           May 31, 1988

-------
EXHIBIT 3-1: DATA MANAGEMENT TOPICS
   THROUGH THE SYSTEM LIFE CYCLE
^PHASE/STAGE
^S.
DATA ^v
MANAGEMENT^.
PLAN TOPIC XVV
Information Need Summary
Dau Steward Organizalions
(Plan)
High Level Dau Requirements
- Conceptual Modeling
Dau Documentation Plans
Responsibilities
Life Cycle Methodology/Tool Plans
Physical Dau Flow Plan
Definition Stage
— Logical Modeling
- Detailed Requirements
Dau Quality Assurance Plans
- Monitor Dau Quality
Data Security Requirements
— Strategy
Design Stage
- Physical DB Design
Dau Conversion Strategy
Testing Support Plan
Dau Testing Strategy
Development Stage
— Programming Support
Implernenutkxi.Slage
— Testing Support
— Production Cutover Plans
Production Stage
— DB Support Plan
- Production Dictionary Support Plan
- Backup, Recovery, Restart Plans
(Procedures)
- Custodial Responsibilities
Evaluation Stage
- Support Evaluation
— Response
Archive Stage
- Tata Base DDL. Metadata, and
Dau Disposition Plan


|
i
2
C












'




















fc
1
8

c

c
c

c
c

c
•























1
1
a

R

R
R

R
R

R

C
C





















z
S
o

R


R

R
R



R
R


C
C
C
c














i
1
§
a

R


R

R
R



R
R


R
R
R
R
C





.






5
P
I
i
-

R


R

R
'R



R
R



R
R

R


C










z
I
§
£




R

R




R
R

•







R




C


C


z
|
1
£













•













R






1
5
*































R

                   I.ERFNP;

             C = CREATE  R = RERNE/REVISE


                    16

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING  THE SYSTEM LIFE CYCLE


    — Entity Identifiers  (Candidate  Keys)
         -  Data  elements used  to  uniquely identify  each entity  (e.g., Employee
         SSN)

    ~ Conceptual Data Model
         - Develop a graphic representation of entities  and  their relationships

    — Likely Sources of Data (by Entity)
         - Identify the likely functions and organizations providing  data

    — Validation of the Conceptual Data Model
         -  Document  your  plan  to  validate that  each  entity  in the high-level
         information  flow  is  in  the  conceptual  data model.  The high-level
         information flow is done early in the-jConcept  Phase, and is  included in
         the System Concept Document.

    — Data Distribution Plan (by Entity)
         -  Record  the  use  of  each  entity in two 2-dimensional matrices, by
         function and by distribution/location.

    -- Information Collection Burden
         -  P.Ian  for any additional  information collection  burden  hours  imposed
         by the solution to the information need.

3.2.3    Data Documentation Responsibilities

    Produce and document plans to allocate responsibilities  for documenting data
requirements, data designs and physical data structures.  Look  for plans  done by
earlier  projects  of  the  same scope, since this can  save  you time  and  effort.
Include  the- data objects to be documented, the attributes  to  be documented and
the  functional  areas  responsible for creating the documentation.   Also detail
the  functional support available to help people create and  record  metadata, and
the   media  to  be  used  for  recording  dictionary  documentation.    Prepare
maintenance  procedures  for  dictionary documentation,  in coordination with the
project's configuration management plan.

3.2.4    Life Cycle Methodologies and Tools

    Record the methodologies and automated tools you will  use during  the  project
to  support  data  management activities.  Be sure to include your explicit plan
for  managing  the flow of metadata (information about  data) between  methods and
tools  through  the  life  cycle.    These  tools  might  include automated data
dictionaries,  data  modeling  and  data  design tools, and  data base management
systems.

3.2.5    Plan For Physical Flow Of Data

    Record  your  plans  for  the  physical data (data  sets) flowing  through the
system.    This  is  particularly  important when automated  data is added to the
system  from  external  sources,  or when multiple physical  data sets need to be
updated  from  a  single source of data.  Later, timeliness  and error correction
procedures should be developed.
                                       17

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


3.3 Definition Stage Topics

    Definition  Stage  activities  and  products are especially important,  since
they  define  a  system's  detailed data requirements.  Chapters 4 and 7 contain
more information about these activities.  Topics covered should Include:

         Interview Plans
         -  Determine  and record the name of the interviewees, functional  areas
         to  be  the  subject  of  interviews, and relationship between the data
        - requirements interview and the functional requirements interview.   (You
         may  want  to  combine  these interviews.)  Also record the information
         needed from the interviews in a pre-written interview guide.   After the
         interview,  follow-up  with  a  written  summary  of the results to the
         interviewee.

         Data Analysis By Process
         -  Determine  and  document  your plan to identify and analyze the data
         elements  required  by each process during the functional analysis (see
         SLCM guidance).  This analysis plus the logical data model  will produce
         the detailed data requirements for the project.

         Entity Normalization
         -  Determine  and  record the methodology you will use to normalize the
         entities  required  by  each  process.  Normalization is the  process of
         reducing  a data entity to its most basic form, removing repeating data
         elements,  data  elements not dependent upon the key of the entity, and
         data  elements  dependent upon the key of other entities.  Your plan to
         perform data analysis should be documented as part of the data analysis
         topic mentioned above.

         Conceptual Data Model Revision
         -  Determine  and  document  your plan for updating the conceptual data
         model  as new entities are determined.  The revision should be on-going
         during the Definition Stage, not held until the end of the stage.

         Logical Data Model
         -  Determine  and  document details of your plan to create logical data
         models  for each functional process, and for the project as a whole.  A
         logical   data  model  is  a  graphic  depiction  of  the  logical,  or
         programmatic,  data  needed  to support an organizational mission.  See
         Chapter  4 for more information.  Once you have a draft of your logical
         data model, hold group review sessions (walkthroughs) to validate  it.

         Requirements Data Dictionary
         -  Determine  and  document your plan for recording metadata describing
         data entities and data elements in the requirements data dictionary. Be
         specific  about  responsibilities, activities, and external support you-
         will need.

         Data Flow/Logical Data Model Validation
         -  Document your plan to validate the data elements in the logical data
         model  against  the  data elements in data flows prepared by the system
         analysts.


                                        18

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM LIFE  CYCLE


3.3.1    Data Quality Assurance

     Much of the improvement in data quality that OSWER requires  will occur  as  a
result  of  following this practice paper's approach  carefully.   However  if  your
system  users  want  to  measure  the  quality  of the  data  in  their   system
periodically,  document  your  data  quality  assurance  plan.  If  you  establish
quantitative  measures of quality for specific data elements, or  combinations of
data  elements,  record your plan for measuring your  data bases'  ability  to  meet
these measures.   Include the following information:

     —  Responsibilities
         - What organization is responsible for monitoring data quality
         - Who will monitor or audit data quality
     —  Data Quality Objective Monitoring Plan
         - Objectives of the data quality assurance plan
         - What data will be monitored
         - How will it be done and how often
         - How will you resolve problems that are raised
         - Who will monitor problem resolution

3.3.2   Data Security Requirements and Strategy

     include your plan for identifying data security  requirements,  recording the
requirements,  and  implementing them.  Responsibility for identifying  sensitive
data  and  protecting  the  data  must  be  detailed  in accordance with the data
stewardship  roles  of  the  project.  Details of data security requirements are
recorded in the Security Manual during the Development Stage.

3.4  Design Stage Topics                                         ,

     Create  and  document your approach to Design Stage data management topics.
Topics should"  include:
                                                               i
3.4.1    Logical Data Model Revision
         - Detail your plan for revising the overall  logical data model.

3.4.2    Physical Data Base Design (Data Design)
         -  Document  the method you plan to use to prepare a physical  data base
         design  (data  design)  for the project.  The physical data base design
         you   prepare  will  be  used  to create the physical data structures to
         support your system.  See Chapter 6 for more information on this topic.

3.4.3    Design Data Dictionary
         -  Enter - your  plan  for  recording  information   (metadata)  about the
         physical  data  base  design in the design data dictionary.  Include an
         outline  of  what  metadata  can  be  copied from the requirements data
         dictionary.

3.4.4    Role  of the Data Custodian
         -  Identify  and document the data custodian for data in the data base.
         With  distributed  data  bases  you  will  have multiple custodians, so
         careful  planning is needed. Record this information in the design data
         dictionary during the Design Stage.


                                       19

-------
                                                                    OSWBfDIRECTIVE #3023.005

         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


3.4.5    Data Conversion Strategy and Plan

         If  existing  data  will  be used in the new system, include details of
         your  plans  for  data  conversion  and  the support of data conversion
         activities.  Information to be included are:

         Sources
    ~   Media
         Load Programs Required (automated procedures)
    —  -Edit Criteria Related to Quality
    —   Validation Plan

3.4.6    Data Testing Strategy

    Document  your  general  strategy  for  testing  data  base  performance and
functionality  to  prove  you have a viable data base design that meets detailed
data  and  system  functional  requirements.    Most  functional   test  planning
information will be stored in the System Test Document.

3.4.7    Testing Support Plan

    Record your detailed plans for supporting testing from the Development  Stage
through the Production Stage.  Information includes:

         Kinds of Physical Test Data Bases Required
         Test Data Provision
         Performance Validation Plan
         Projected Testing Support Needed

3.5 Development Stage Topics

    Your  approach  to  Development  Stage  data  management  activities  should
include:

3.5.1    Data Structures for Programming Support
         -  Record  your  plan  to support software program development and unit
         testing  here.  Coordinate your plan carefully to avoid problems.   This
         section will vary depending upon the data base and programming language
         you use.
                                                              *•
3.5.2    Data (structure) Revision Approach
         - Document the details of the change control procedures you will use to
         support  changing  and  adding  data structures to the data base during
         this  stage.   Refer to the project's Configuration Management Plan for
         guidance  concerning the overall framework you will use.  Also document
         your approach to updating the design data dictionary to keep abreast of
         all changes.

3.5.3    Data Backup, Logging and Recovery Plans
         -  Record your plans for backup, logging, and recovery of physical data
         sets  stored  in  the  data  structures  your  project  creates.   It is
         suggested  that  you  document your plans for test data bases, and your
         plans for the production versions of data.


                                       20

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM  LIFE  CYCLE


3.5.4    Data Documentation and User Training  Materials
         Ensure that data management concerns  are included  in  all documentation,
         not  just  the  data  dictionaries,  and  that  user   training material
         includes  an  emphasis  on  data  creation,  collection,  validation,  and
         quality assurance issues.

3.6 Implementation Stage Topics

    Your approach to Implementation Stage data management topics  covered  in your
plan should include:

3.6.1    Testing (See Acceptance Test Plans)
         -  Include  plans  for  supporting  integration  testing and acceptance
         testing with the data bases that have been built.

3.6.2    Cutover Plans
         -  Record  the  activities  the  data base administration function must
         carry  out  to  support cutover to the production system.   This  cutover
         will  usually  involve  unloading  test  data,  securing the production
         version  of  software, and loading 'data that has been converted  from an
         existing system.

3.7 Production Stage Topics

    Topics covered should include:

3.7.1    Data Base and Metadata Management
         -  Be  sure  to have procedures, and.plans for supporting the production
         data  base  and  keeping the production data dictionary accurate.  Also
         describe procedures to keep the design and production data dictionaries
         in synchronization when enhancements occur.

3.7.2    Support of Configuration Management
         -  Coordinate  data  management  with  the  configuration  plans of the
         project.    Refer  to  the  project's Configuration Management Plan for
         details  of  the  project's  approach.  Pay particular attention to the
         updating  of  data  dictionary  information,  and  control  of  changes
         requested for data bases.

3.8  Evaluation Stage Topics

     Topics covered should include:

3.8.1    Audit and Evaluation Support Plan
         -  If  data management activities are required to support the audit and
         evaluation process, document the planned activities.

3.8.2    Response to Evaluation Report
         - If data management related actions are required in response to points
         of an evaluation report, document the planned actions.

3.9  Archive Stage Topics

     Topics covered should include:

                                       21

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


3.9.1    Data Base Data Definition Language and Metadata Disposition
         -  Document  plans  to  archive  or  pass to another data base the  data
         definition  language  statements that create the data structures  in the
         data base, and the metadata in the production data dictionary.

3.9.2    Data Disposition
         -  Document plans to a-chive or pass to another data base the data  sets
         (data) in the data base.

3.9.3  "  Cutover Procedures
         -  Document  plans for the data base administration function to support
         the cutover from production to archiving or another system.
                                        22

-------
         PRACTICE  PAPER FOR DATA MANAGEMENT  DURING THE SYSTEM LIFE CYCLE



                                    Chapter  4

                            DATA MODELING  ACTIVITIES


4.1  Introduction

     This  chapter  presents  the  key concepts of data modeling  and  lists  topics
outlined 1n the Data Management Plan that  pertain to  data modeling.

4.2  Overview

     Data  modeling  activities relate the data requirements of a  project to a
programmatic, or end-user perspective.  These activities  provide essential input
into  the statement of requirements for a  project.  There are two levels of data
modeling that you  will perform during a project:

     o   Conceptual Data Modeling
         - a broad look at data requirements

     o   Logical Data Modeling
         - expansion of the conceptual model to include  detailed requirements

     You  perform conceptual data  modeling during the Concept  Phase, and  logical
data  modeling  during the Definition Stage.  In both cases, the data models are
validated  by checking their completeness  against data flow diagrams prepared by
systems  analysts,  and  by  reviewing  them  with   end-users.    Conceptual data
modeling  contributes to the preparation of high-level  data requirements  for the
System  Concept, and the logical  data modeling contributes to  the preparation of
detailed  data  requirements  that  are  documented   in   the  requirements  data
dictionary.

4.3  Conceptual Data Modeling

     Conceptual  data  modeling,  the first step in  data modeling, begins  during
the  Concept Phase.  A conceptual  data model shows  the kinds of  data that  should
be  stored  for  system  users,  and  the relationships  among  the data.  Often a
conceptual data model  is used to coordinate the data requirements of one  project
with  the  requirements of other projects by providing a framework for assigning
data elements to specific, high-level data entities.

     The  data  identified  during  conceptual  data  modeling   are  called data
entities.  A data entity is a person, place, thing,  concept or  event about which
OSWER  needs  to  obtain  data.    Facility,  Determination,  and Permit  are all
examples of data entities.

     You begin conceptual data modeling by preparing  a list of  data entities for
the  project.    The   11st can be  prepared by examining  documents describing the
processes  to  be  automated, by interviewing programmatic staff directly, or by
working, in conjunction with systems analysts during  their initial interviews of
users.  Exhibit 4-1 is an example  of a data entity  list.
                                       23                           May 31, 1988

-------
    EXHIBIT 4-1: ENTITY LIST
          EXAMPLE
         Award
       Employee
       Institution
      Organization

        Proposal
•«W-
Part of high level data requirements,
             24

-------
PRACTICE PAPER FOR DATA MANAGEMEHT DURING THE
                                                              ,.*».. „.
     Then,  each  data  entity  is  defined  in programmatic terms,  and the data
       s that can be used to uniquely identify each occurrence of each entity
     identified.    Record the data entity definitions carefully, since they are
needed for the System Concept document, and for the requirements data dictionary
(Definition Stage).

     Next preoare a picture of the entities, the relationships between entities,
and  the  data  elements  that  uniquely  identify each entity (sometimes called
candidate  keys).   Exhibit 4-2 is an example of such a picture.  These pictures
are sometimes called entity-relationship diagrams.

     Finally,  review  the  information  flows, or high-level data flow diagrams
prepared  by  systems analysts to ensure that data entities representing all the
data  in the data flows and stores on these flow diagrams are represented in the
conceptual  data  model.    Additional data entities are defined, added, and the
entity-relationship diagram is updated until it is complete.

4.4  Logical Data Modeling

     Logical  data  modeling  is  a vital activity during the development of any
data  base,  since  it  includes  analysis of programmatic data requirements and
provides  the  starting  point for physical data base design.  Modeling includes
depicting   the  data  required  for  programmatic  functions  graphically,  and
validating  the  accuracy . of  the  requirement with the users of the data.  The
logical  data  model provides a clear, accurate description of data requirements
that  physical  data  base designers (see Chapter 5) use to begin design of data
bases.

     Logical  data  modeling  is  performed during the Definition Stage, and the
logical  model  is  maintained throughout the life cycle.  Logical data modeling
builds  upon the conceptual data model diagrams and entity definitions that were
prepared during the Concept Phase.  This model .extends the conceptual data model
by  identifying  the data elements required to describe each data entity.  While
the  conceptual data model might contain the data entity "Employee" and the data
element  "Employee Number" that uniquely identifies the entity, the logical data
model  would  include  other  data  elements, often called attributes, needed to
describe  "Employee".    Examples of such data elements include "Employee Name",
"Employee Home Address", and "Employee Birth Date".

     Perform a detailed data analysis of the data flows and stores documented by
systems analysts during the Definition Stage to expand the conceptual data model
into  a  more  detailed,  logical  data  model  by  adding and defining the data
elements required for each entity.

     Construct a logical data model diagram that includes data entities and data
elements  needed by all processes.  Then, "normalize" data entities to eliminate
logical  redundancy,  so  that your "normalized" logical data model will contain
each  data  element  in  only  one  place.   See Exhibit 4-3 for an example of  a
logical data model.

     When  you normalize, the number of data entities will increase dramatically
as  you  are  forced  to  identify more specific entities that portray sub-types
("Manager",  a  sub-type  of  "Employee") and roles ("Assigned Employee") of the
entities  you  had  in  your  conceptual  model.    Don't  be  alarmed  by  this

                                       25

-------
     EXHIBIT 4-2:   CONCEPTUAL DATA MODEL
                              EXAMPLE
Entity
Unique Identifier
(Candidate Key)
  INSTITUTION
   (External)
                                     Authorizes
                                                        Relationship
                                    PROPOSAL NUMBER
           PROPOSAL
This conceptual data model describes the relationship between data entities composing the high level
data requirements for a proposal award function. Of particular note is the fact that one proposal can
result in many awards.  Other relationships in the model are:

                One  Institution
                One  Employee
               , One  Organization
                One  Proposal is

                One  Institution
Submits
Reviews
Authorizes
Funded by
Receives
Employs
Many
Many
Many
Many
Many
Many
-Proposals
Proposals
Awards
Awards
Awards
Employees
                        LEGEND
                                      Indicates One Entity
                                      Indicates Many Entities
                                      26

-------
                  EXHIBIT 4-3:  LOGICAL DATA MODEL
                                       EXAMPLE
                           PROCESS: PRINT A SUMMARY OF PROPOSALS
                        FUNDED LAST MONTH BY SUBMITTING INSTITUTION
  RELATIONSHIP  ENTITY NAME   DATA ELEMENTS (ATTRIBUTES)
ro

	 ww


£ FKUFUSAL

g£ PROPOSAL REVI

^£ KMHiWKK
	 +
f ORfiANI7ATION
k.b.

£^ AWAKU
	 fe.
                                       ERQJ, PRO NAME, PRO SUBMIT DATE, IINS   IDI

                                       P'RO * +- ™P SSN # + RRV #, REV DETERMINATION,
                                       REV START DATE, REV END DATE
RMP SSN #f BMP NAME. EMP PHONE. JORG NAME]

QRfiJD, ORG NAME
                                       AWDff, AWD AMT, AWD DATE, lORG ID I, iPROfll JlNS Id

                                       INS ID. INS NAME
     This is a logical data model for the process of printing a summary by submitting institution of funded proposals. The entities
     here are drawn from a conceptual data model that covers many processes. Once this logical model is normalized, any new
     entities found are added to the conceptual data model, and used to update the overall logical data model.
                        + Indicates Cuiu .deflation
    LEGEND

  Entities in bold  , Separates Data Elements
                                Primary Kcvs Underlined   (Secondary Kcvs in Boxed

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


proliferation  of  data  entities, it is a sign of your success at "normalizing"
entities.    This proliferation of entities is caused by your being increasingly
more specific in describing and modeling your project's data requirements.

     Validate  that  you  have  identified  all  data  entities  and elements by
examining the systems analyst's data flow diagrams again to ensure that all data
elements  in  the  data  flows  and  data stores have been accounted for in your
logical  data  model.  End-users  should  then review the model through a formal
process, and "sign-off" on the logical  model's accuracy.

     Finally, add all the descriptions  of the data entities and data elements in
the  logical  model,  as  well  as  their  definitions, to the requirements data
dictionary.    Contact the Information  Management Staff of OPMT to obtain a copy
of  relevant  OSWER documentation standards.  Enter data stewardship information
into  the  dictionary (see Chapter 6).   The requirements data dictionary an'd the
logical   data   model  diagrams  taken  together  comprise  the "detailed  data
requirements for the project.

4.5  Data Management Plan Topics Related to.Data Modeling

     Topics  in  the  Data Management Plan that cover data modeling are included
below.  Methodologies and tools should  be selected before data modeling begins.

o    Concept Development

     ~  Entity List
     —  Entity Definitions                                .
     —  Entity Identifiers           .               '
     —  Conceptual Data Model
     —  Likely Sources of Data
     —  Information Flow/Data Model Validation
     —  Data Distribution Plan
     —  Information Collection Burden

o    Definition Stage
     •
     —  Interview Plans
     —  Data Analysis By Process
     —  Entity Normalization
     —  Conceptual Data Model Revision                      *
     —  High-Level Data Requirements Revision
     —  Logical Data Model
     —  Data Flow/Logical Data Model Validation

o    Design Stage

     —  Logical Data Model Revision

o    Data Security Requirements and Strategy
     —  Sensitive Data
      (Identified During Data Analysis and Modeling)

o     Life Cycle Data Management Methodologies and Tools


                                       28

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
                                    Chapter 5


                      PHYSICAL DATA BASE DESIGN ACTIVITIES
5.1  Introduction
     This  chapter  presents an overview of physical  data  base  design  activities
and  lists  the topics in the Data Management Plan that  pertain to  physical  data
base  design.   Physical data base design activities  support  the storage of  data
needed  by  a  system's  users.    Contact the National  Data  Processing Division
(NDPD)  for  a  copy  of  the  procedures  for  implementing  data  bases on  NDPD
equipment.

5.2  Overview

     Data design activities have been a part of system development  practices for
some  years  now.  The product of these activities has been data structures, for
example  physical  record types, that support a system's software programs.   The
development  of physical data base structures without data modeling has resulted
in unstable designs that increase maintenance costs,  and limit  the  usefulness of
data stored in these structures.  So don't begin your data management  activities
by doing a physical data base design.

     Physical  data  base  design activities begin during the Design Stage after
the  logical data model is completed.  Information contained  in the logical  data
model  and  the requirements data dictionary is used  as a starting  point  for the
design  process.  The designer transforms the logical data model into  an  initial
physical  data  base  design  that  can be implemented in a data base  management
system.  Data definition language statements are produced during the Development
Stage  to  provide  common  data  structures  for  programming   and unit  testing
support.    During  the  Development  and  Implementation  Stages,  the design is
altered  to  meet  the  performance  requirements  of  the  system, and the data
definition language statements are also revised.

5.3 The Physical Data Base Design Process

    The  physical data base design process must include participation  by  someone
with  an expert knowledge of the DBMS being used.  You may decide to do physical
data  base  design in a single step, but if you have  a large  and complex  system,
you  may  want  to  plan  to  follow  the steps outlined below.  There are three
significant  products  in  the  physical  data  base  design process that  you may
produce to ensure thoroughness:

         o  Data driven physical model
         o  Process driven physical model
         o  Location driven physical model

These  three products are the focal point of the process outlined below.   Please
note  that  in some circumstances you may combine two or three of the  models and
produce a single model instead.

                                       29                    '.       May 31,  1988

-------
                             •                                     OSWtH

         PRACTICE PAPER FOR  DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


5.3.1  Step 1:     Data Driven Physical Model

       Begin  your physical  data base design process by taking your logical  data
model  and  transforming   it to  fit  the structure of the data base management
system  (DBMS)  you  are   using.    If you are using a DBMS that can implement a
relational  data  model  directly,  then this will not require much work  at  all,
since your normalized logical data model will be structured in relational  terms.
With  a relational DBMS the  logical model is the data driven physical model.   If
you use a relational DBMS  and have no requirements for high-volume processing or
distributed processing, your physical data base design 1s completed.

       If  you  are  using   a  DBMS  such  as FOCUS or SYSTEM 2000 that  does  not
support  the relational data model, you will have to transform your logical  data
model  into  the  hierarchical  or  network  structures  which these other DBMS'
support.  This activity involves allocating data elements from your logical  data
model to more than one physical record or segment, depending upon the DBMS being
used.    Assign  an  individual  with expert-level knowledge of the DBMS  package
being used to perform this task.  Ask the DBMS vendor if they can provide design
examples or material for you.

5.3.2  Step 2:     Process Driven Physical Model

       Review  the  transaction  volumes  and service requirements of particular
software  programs  to  determine  if  high volume or very fast response  time is
required for some transactions.  If so, alter your data driven physical  model to
place  the  data  elements needed by the high volume or high service requirement
programs  together  in one physical record.  This introduces phys-ical redundancy
that  may  be  justified   by .your  application.   Schedule a walkthrough of  the
process  driven  physical  model with systems analysts and programmers,  the  data
base designer, and the project's data administration staff.

5.3.3  Step 3:     Location  Driven Physical Model

       If you plan to distribute your data base(s) at more than one location and
run  software  against  the  distributed data, then you will have to modify  your
process  driven  physical  model to account for the replication of data needed to
support  distributed  processing.  This 1s not a trivial activity.  Make certain
that the design team has expertise in the design of distributed data bases using
the selected DBMS package.
                                                              4
5.4 Using Your Physical Data Base Designs

    Normally,  the  process  driven  physical model will be the initial  physical
data base design for your  project.  If you are using a distributed data base for
your project, you will have  to complete a location driven physical model  and use
it  for  your physical data  base design.  Whichever is the case, use the initial
physical data base design  to support the preparation of data definition language
statements  for  your  DBMS  during  Development.  If  no  DBMS  is  used, these
statements will be replaced  by copybooks or copy libraries that can be shared by
programmers.

    Once  the  physical  data  base  design  is  done,  create  the  design data
dictionary  (if you haven't already done so), by copying the definitions for data
elements in the requirements "data dictionary into the design data dictionary and

                                        30

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


adding  technical  information  that  describes  your physical data base design.
Maintain  the  design  data  dictionary  and  transfer, it to the production data
dictionary  at  Implementation  Stage.    See  Chapter  7 .for  more  details on
documentation.

    The  physical  data  model (physical data base design) is created during the
Design  Stage,  and  the  data definition language statements (or copybooks) are
produced  early in the Development Stage to support development and unit testing
of  programs.    These data structures are maintained throughout the rest of the
system life cycle under the control of configuration management.

5.5 Data Management Plan Topics Relating to Physical Data Base Design

    Topics  in the Data Management.PI an that cover physical data base design are
included below.

    Design Stage
         Logical Data Model Revision
         Physical Data Base Design
         Design Data Dictionary

    Development Stage
         Data Structures for Programming Support
         Data Structure Revision Approach

    Implementation Stage
    — Testing Support                          .   •       ..

    Life Cycle Methodologies arid Tools
    (Supports physical data base design activities)

    Data Security Strategy
    (Requirements are supported by the physical data base design)

    Plan for Physical Flow of Data
    (Supported by the physical data base design)

    Testing Support Plan
    (Supported by the physical data base design and DBMS)
    —f   Performance Validation Plan
                                       31

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE




                                    CHAPTER 6

                       DATA STEWARDSHIP RELATED ACTIVITIES


6.1  Introduction

     If. you  are  the  individual  responsible  for data management, you should
create  a  plan  to  determine  and  document data stewardship for the data your
system will collect.  This plan should be done early in the Concept Phase, since
.several concept activities implement data stewardship.

6.2  Background

     The  Office  of  Solid  Waste and Emergency Response uses thousands of data
elements  while  performing  the  missions  assigned  to  it.  The flow of OSWER
program data is often lengthy and complex, as data is collected at the state and
regional  levels,  then forwarded to OSWER 'for analysis, with the results being
disseminated to Congress, other agencies, the states, and regions.  Managing the
complex  activities,  responsibilities  and relationships which arise from these
data  flows requires a method of determining which organizations involved in the
data flows are responsible for which data-related activities.

6.3  Guidance Objectives

     The  data  stewardship guidance presented ..in this chapter has the following
objectives:

     a.   Facilitate  the  assignment  of  responsibilities for data definition,
     collection,  processing,  storage, and use, when systems and data bases are
     being built.

     b.   Ensure data meets mission requirements by assigning accountability for
     data stewardship.

     Co   Ensure  that  data  definition,  collection;  processing,  and storage
     methods within OSWER systems conform to applicable guidance.
                                                             /
     d<   Facilitate   data   sharing   and   reuse   by  clarifying  roles  and
     responsibilities   involved  in  the  definition,  collection,  processing,
     storage, and use of data.

6.4  What Is Data Stewardship?

     Within OSWER, organizations may share data, but do not individually own it.
OSWER  defines  data  stewardship  as  the  functions and responsibilities of  an
organization  that  exercises programmatic control over data on behalf of OSWER.
Organizations  that  require  data to be collected, processed, stored .or used  in
support   of   OSWER1s   mission   have  stewardship  responsibilities.    These
responsibilities include ensuring that:
                                       32                           May 31,  1988

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


         1)    Data   is  clearly  defined  and  documented  in  compliance  with
              established directives.

         2)    Data that is collected  is  of sufficient quality to support OSWER's
              missions.

         3)    Only data relevant to OSWER's missions -is  collected.

         4)    Data is reused wherever appropriate within OSWER.

         5)    Data  management  practices  for  data  under  the  organization's
              stewardship conform to  OSWER guidance.

6.5  Which Organization Is A Data Steward?

     The  data  steward role is separate and  distinct from the role  of  a project
manager.    Data  stewardship  assignment parallels  OSWER  program management
responsibility.    The  same  organization with overall  responsibility  for  OSWER
program  mission  performance  is responsible for  ensuring that the  quality data
required  to  support  the  mission is defined, collected, processed, stored and
presented  in a timely and cost-effective manner.   If a  project involves support
of  more  than  one program, then multiple data stewards may be involved for the
data elements used by the project's system.   Exhibit  6-1 provides  an overview of
the data steward role.

6.6  What Are The Rights and Responsibilities of  a Data  Steward?

     Data  Steward Rights.  The data steward  organization appoints  an individual
to  execute  its  rights within the data stewardship  guidance.  The  data steward
individual sanctions or approves the definition,  collection, processing, storage
and  use  of data under his/her stewardship.   This individual  also  has  the  right
to designate the data definer, the data collector and the custodian (see section
6.7).

    • Data   Steward   Responsibilities.     The  data  steward  organization  is
responsible  for  ensuring  that  all  the  functional   data roles  are  performed
adequately  to  provide  data  of  sufficient  quality   to support  OSWER program
missions.    Other  responsibilities include ensuring that only data relevant to
OSWER missions is collected,  that data is reused  wherever appropriate, and that
data  is  clearly  defined  and  documented   in  conformance   with  EPA  and  OSWER
guidance.    The  steward  organization  appoints   an  individual  to execute its
responsibilities within the data stewardship guidance.

6.7  What Are Functional Data Roles?

     Exhibit 6-2 summarizes the function of  each  data role.  A longer discussion
of each functional data role follows.

     a.   Definer.

          The   organization   or   function  responsible  for determining  and
documenting  the  name,  description,  and  other attributes of data required to
support  an  OSWER mission is the data definer.  The  data definer  prescribes the
derivation  rules  and  the  formats to be used for data derived  from other data

                                       33

-------
               EXHIBIT 6-1:  THE DATA STEWARD  ROLE
        Data Steward
        Organization
   Appoints
CO
Designates
                                                                    Definer
                                        Collector
        Data Steward
                                                                    Primary User
                                        Custodian
                             Sanctions
                                                                    Ancillary
                                                                    User(s)
 Rights:          Sanction definition, collection, processing, storage, and use of data.
                 Designate data definer, collector, primary user, and custodian.
 Responsibilities:  Ensures functional data roles are performed adequately to provide data of
                 sufficient quality to support OSWER program missions.

-------
            EXHIBIT 6-2: FUNCTIONAL  DATA ROLES
      ROLE
                     FUNCTION
CO
in
       Definer
      Collector
    Primary  User
     Custodian
     Ancillary
      Uscr(s)
                       Defines data requirements, meaning, content, and documents the needed data in     "N
                       compliance with OSWER guidance.                                         J
                       Collects data required by OSWER missions.
                       Uses data to perform mission functions. The primary data user is often the
                       data definer, as well. Supports testing during the systems life cycle.
                       Responsible for supporting the storage and processing of data. Ensures the physical
                       integrity of data used to support OSWER missions while providing access to the users.
                       Implements changes.
Uses data defined and stored to support the missions of others. The Ancillary user
requests appropriate approval from the Data Steward for accessing and using the data
sought.                              .

S
t
\

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


elements.  The data definer may or may not be the same person or organization as
the collector.  Usually, the data steward designates the primary user(s) of data
designated  as  the  data definer.  When multiple functions or organizations use
the  same  data  to  support  important  OSWER  program  functions, a joint data
definition  effort  will  be  organized.    The  data definer is responsible for
ensuring  that OSWER and EPA data administration standards are met when defining
and documenting data.

     b.   Collector.

          The  organizations  or functions responsible for collecting the values
of  data  required by an OSWER program are called the data collectors.  The data
steward  organization designates data collectors to direct and manage collection
of  data  for an OSWER program, after consulting with the collector organization
to  ensure  it  can assume this role.  The data collectors may not terminate the
collection of data without the authorization of the data steward organization.

     c.   Primary User(s).

          The  organization  or  function with the most important requirement to
collect,  store  and  process  data to perform a current or future OSWER mission
function is the primary user.  Usually, the data steward organization designates
the  primary user as the definer of the data the organization requires.  Primary
user  organizations  also support user testing of systems during the life cycle.
If  it  .is  impossible  to  select a single primary user organization from among
several  users  of  the  same  data,  then  a  joint data definition effort will
probably  result.   The primary user organization is often, although not always,
the data steward organization.

     d.   Custodian.

          The organization responsible for storage and processing of data is the
data  custodian.  The functions carried out by the custodian include those which
have  traditionally been performed by ADP organizations.  That is, the custodian
has  physical  custody  or  direct  control  of the data, software and other ADP
components  used to store, process, communicate and present data.  The custodial
role ensures the physical integrity of data, and safeguarding the storage media.
During  the  life  cycle, the custodial role may be assigned to the project team
until  the  data  base (and system) is turned over to the operating organization
during the Implementation Stage.  As small computers have become more common, it
is  not  unusual  for  custodial  responsibilities to be split between different
organizations,  or  to  be  assumed directly by non-ADP professionals within the
primary user organization.

     e.   Ancillary User(s)

          The  ancillary  users  use data to perform OSWER functions, and report
results  to  management, the Congress, and to others outside the agency.  Unlike
the  primary  users,  ancillary  users must rely upon others to define and allow
them access to the data.
                                       36

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE  CYCLE


6.8  Implementing Data Stewardship

     During  the  Concept Phase, you should contact the OSWER Data  Administrator
and  discuss  the  scope  of your project's application and the general types  of
information you anticipate the application will need.  If you and the OSWER data
administrator  determine  that  multiple  data  steward  organizations will  be
involved  in  a project, plan to involve all of these organizations beginning  in
the  Concept  Phase.    Since  implementation of stewardship at the data  base  or
project  level  is  not  feasible  when  multiple data steward organizations are
involved,  record  the  steward  organization  for each data entity you  identify
during  Concept  Phase.   While there may be multiple data stewards for  the data
collected  by  a system, there can be only one data steward per element  of data.
Enter  this  stewardship  information in the requirements data dictionary during
the Definition Stage.  Ensure that data definers, primary users, collectors, and
custodians are recorded for all data elements.

     As  noted in Exhibit 6-3, when one organization defines, collects,  and uses
a  file, that organization is the probable data steward.  However,  when  you find
data  definition,  collection,  and use split between multiple organizations you.
face  a  more difficult problem when determining stewardship.  Line 2 of Exhibit
6-3  provides an example of multiple organizations collecting and using  the same
data  file.    When  this occurs either of the primary user organizations may  be
selected  as the data steward, but not both.  Data stewardship of data cannot  be
shared  between  organizations.    A  single  data  steward organization must  be
selected for clear accountability.

     If  your project is developing a high-impact system, you are likely to find
that  several  data  steward  organizations will be involved.  For instance, the
application  may  collect  enforcement data at the request of OWPE, and  clean-up
data  at  the  request  of  OERR.    Be  sure that both offices understand their
stewardship  responsibilities,  and  work  with  them  to  provide the data they
require.

     Use  data dictionaries to save time when recording stewardship information.
For  instance,  if  one  steward organization sponsors an entire data base, then
record  the  stewardship  information  as  part  of  the  data  base  dictionary
documentation,  and  use  the  dictionary  reporting  facilities  to  link  this
information to all data elements in the data base.

Data Stewardship Implementation Activities

Step 1 - During the Concept Phase identify the organizations that will likely be
the data steward organizations for the data your project will require.

Step 2 - Have the data steward organization appoint a data steward individual  to
assume data stewardship responsibilities.

Step  3  - Work with the person appointed to determine the primary data user and
assign  a data definer or definers for the project.  Consult with the OSWER data
administrator.  Involve the data definer(s) in identifying data entities as your
project team performs conceptual data modeling.
                                       37

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


Step  4  -  After you have completed your data entity 11st (see Chapter 4),  have
the  data steward assign specific responsibility for defining each data entity's
meaning in mission terms.  This same data definer will also define data elements
to  describe  the  data  entity more fully when the Definition Stage is reached.
Record  the name of the data definer and data steward organization for each  data
entity in the System Concept and in the definitions of data entities you create.

Step  5  -  During  the  Definition  Stage, record the data steward and the  data
definer  organizations  for each data entity in the requirements data dictionary
and the detailed data requirements.  Design your data dictionary entries so  that
dictionary  users  can learn the data steward and definer for each data element,
too.

Step 6 - During the Definition Stage, determine the function and organization to
collect  values  for  each  data  element;  record  the  data  collector  in the
requirements data dictionary.

Step  7  -  During the Design Stage, have the data steward name the organization
that  will support technical operation of the system and data base once it is in
a  production mode, and designate that function or organization as the custodian
for  all  data elements in the data base.  Record this information in the design
data  dictionary along with the steward and definer information you added to the
requirements dictionary earlier.
                                        38

-------
                                                 EXHIBIT 6-3
                                DETERMINING  DATA  STEWARDSHIP
                                 REVIEW FUNCTIONAL DATA ROLES TO DETERMINE PROBABLE DATA STEWARDS
      DATA OBJECT
DATA

DEFINER
DATA

COLLECTOR
PRIMARY

USERS(S)
NON-PRIMARV

USER(S)          CUSTODIAN
                                                                                                    PROBABLE
                                                                                                    DATA STEWARD
      File I
                         ORG A
                                        ORG A
                                                       ORG A
                                                                      ORG none
                                                                                     ORG C
                                                                           ORG A
CO
      File 2
      Data Element  V
                         ORG A.B
                         ORG B
               ORG D
                                        ORG D
                              ORG A.B
                                                       ORG B .
                                                                      ORG D.E
                                                                      ORG A.O
                                                            ORG C
                                                                                     ORG D
                                                            OHU A or B

                                                            (not tiotli)

                                                            ORG B
      Data Element Z
                         ORG O
                                        ORG 0
                              ORG D
                                                                      ORG A
                                                            ORG C
                                                            ORG D
     Data Base X
                         ORG A
                                        ORG A
                                                       ORG A
                                                                      ORG B.D
                                                                                     ORG C
                                                                                                    OKG A
                                        ORG A = OSWER program office A
                                        ORG B = OSWER program office B
                                        ORG C = EPA data processing organisation C
                                        ORG D = EPA regional office 0

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
                                    CHAPTER 7

                          DATA DOCUMENTATION ACTIVITIES
7.1  Introduction
     This  chapter  describes  the  essential data documentation activities that
must be performed during the system life cycle.  While documentation is  required
of other life cycle products, such as the System Concept,  this paper covers only
the  essential  documentation  of data requirements,  physical  data base  designs,
and  production  data  base  structures.  The documentation must be completed to
support later phases of the life cycle, reduce maintenance costs, and provide an
audit  trail  from requirements to production data bases.   Careful management of
this  documentation  will  allow  you  to track your  project's data requirements
through to the production data base.

7.2  Data Documentation Products

     The OSdER Systems Development Life Cycle Management Guidance specifies that
project  teams  document detailed data requirements,  physical  data base  designs,
and  production  data  bases  in  data  dictionaries.   Each of these three data
dictionaries  serve different purposes.  While paper  or a  single software  system
can  hold  all  three  of these dictionaries, the dictionaries should be created
discretely.

        Dictionary Type                               Role

   Requirements Dictionary       Contains High Level  and Detailed Data
                                 Requirements

   Design Dictionary             Contains Physical Data Base Design

   Production Dictionary         Contains Production  Data  Base Design

7.3 Requirements Data Dictionary

    The  requirements  data  dictionary  is produced  by data modeling activities
that  take  place  during  the  Concept  and  Definition  Phases of the  project.
Metadata  recorded  about  data  entities and data elements include their  names,
programmatic  definitions,  purpose  in  the  OSWER programs,  data steward, data
definer, and source.  Precise guidance regarding the  metadata to insert  into the
requirements data dictionary can be obtained by contacting the Office of Program
Management and Technology Information Management Staff.

    Many  automated  software  development  tools  provide  good support for the
requirements data dictionary.  If automated tools have been used to perform data
modeling, then some of the documentation in the requirements data dictionary can
be  loaded  from  the  software  development  tool  into  the  data  dictionary.
Likewise,  automated  tools  used  to  support  physical data base design should
                                       4Q                           May 31, 1988

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM LIFE CYCLE


contain  metadata derived from your data modeling tool, or from  the requirements
data  dictionary.   If the metadata used in each tool Is based upon the metadata
produced  by  the preceding tool, you will  save time and effort  while minimizing
the risk of inaccurate metadata being used  to produce a data base design.

7.4 Design Data Dictionary

    The  design  data dictionary is initially created in the Design Stage  and  is
maintained  through  the  rest  of  the  life  cycle.   This dictionary contains
detailed  data  documentation  of the physical data base design  (see  Chapter 5).
It  also  contains  programmatic  and  administrative  information  stored  in the
requirements data dictionary earlier.  Programmatic definitions  of  data elements
stored   in  the  requirements  data  dictionary  are  entered   into   this  data
dictionary,  as the physical design information is entered.   If  software  is used
to store and use metadata, then the same piece of software can sometimes  be used
for both dictionaries.  This is a project decision.

    Technical  information  about  the physical data base design is  entered into
the  design  data dictionary.  It describes the design  of the physical data base
structures  and the manner in which these structures are implemented  in  the test
versions  of the data base.  For this reason the design data  dictionary  contains
many  more types of metadata than the requirements data dictionary.   In  addition
to  data  elements,  the design dictionary contains documentation of  data bases,
physical records, segments, data sets (or files), and' keys.   Information  such as
block  sizes,  data  set  allocation, and physical size limits  are documented in
this dictionary.                              .

    If  your  project is using a data base management system  which is controlled
by an active data dictionary, then use the active data  dictionary for the design
data  dictionary.    These active dictionaries contain  only  the  metadata  for the
data  bases  they  control,  and  they can save you time and  money during system
maintenance.

7.5 Production Data Dictionary

    The  production  data  dictionary is initially created  in the Implementation
Stage  to  support system integration and acceptance testing.   Metadata  from the
last  version  of  the  design  data  dictionary is moved to  the production data
dictionary  at  the  beginning  of  the  Implementation Stage.   Exactly  the same
metadata is in the last version of the design dictionary and  the initial  version
of  the  production  dictionary.    The only difference between them is  that the
former describes a data base under development and the  latter a data base in the
final steps of testing and implementation.

    You  should continue to maintain a design data dictionary to support testing
of  new  versions  of the data base throughout the rest of  the  life cycle.  Once
the  production  dictionary  is created, the design dictionary  contains  the same
information  as  the production dictionary, plus proposed changes.  If you use a
data base management system that is controlled by an active  data dictionary, use
the  active  DBMS  dictionary  as the production data dictionary.  The "move" of
metadata  from the design to production dictionary can usually  be done by simply
changing  the  status  indicator  on  those  dictionary objects being moved into
production.


                                       41

-------
                                                                  OSWER DIRECTIVE ff902B.UUa

         PRACTICE PAPER  FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE


    Once  metadata  is   moved  into the production data dictionary it is secured
against  changes  or  deletion.    Entries in the production data dictionary are
never  written-over  or  deleted,  but new versions are created that reflect any
changes.  Old  versions  are  archived  to  create  a  perpetual  history of the
structure  of  the  production  data  base.    The production data dictionary is
maintained  up to the point when the data base is no longer used.  Once the data
base  is  slated for conversion to another data base, or archiving, the metadata
in the production data dictionary is archived along with the production data and
software programs.

7.6 Importance of Metadata Management

    As  technology  has  evolved, data dictionaries have taken on an increasingly
important  role.    Many of  the  most  modern, relational data base management
systems  require  that a production data dictionary be loaded before the DBMS is
used.    Now,  system  and  data  base  design activities are being supported by
automated  software  tools  that  are  also designed around data dictionaries of
their  own.  As the individual responsible for data management for your project,
you  will  have to plan  and monitor your project's collection, use, and transfer
of  metadata  throughout the  system life cycle.  If you fail in this task, you
could  delay the project, increase the cost of the project, or cause a data base
to  be  implemented  that doesn't meet OSWER requirements.  You would be wise to
consult  with  the  OSWER  data  administrator  concerning  the  availability of
metadata  management  software, including the availability of an OSWER-wide data
resource inventory, to support your project.

    Exhibit  7-1 illustrates the use of data dictionaries during the system life
cycle.
                                        42

-------
           DATA  DICTIONARIES  DURING THE SYSTEMS  LIFE CYCLE
                                               INITIATION
                                                 Identify
                                                Information
                                                  Need
                                                                              Define
                                                                            High Level
                                                                               Data
                                                                            Requirements
                     Archive
                     Data &
                    Software
EVALUATIO
                                                                Pre-t listing
                                                                   Data
                                                                  Sources
                                                                                          DEFINITION
                                       Production
                                         Data
                                       Dictionary
                                                                                               Define
                                                                                              Detailed
                                                                                                Data
                                                                                            Requirements
Evaluate
Data Base
                Production
              Data Dictionary
                                                                                   Requirements
                                                                                  DaU Dictionary
                                        METADATA
                                                                               Requirements
                                                                              Data Dictionary
                   Production
                 Data Dictionary
                   (Updated)
PRODUCTION
                                                                             Design
                                                                          Data Dictionary
                    Production
                  Data Dictionary
        Operate
      and Maintain
       Data Base
                                                                                                      Design
                                                                                                     Physical
                                                                                                     Data Base
   Design
Data Dictionary
                                                    Design
                                                    Data
                                                  Dictionary
                                                              Design
                                                          Data Dictionary
                                                             (Updated)
                                           .  Production
                                           Data Dictionary
                                                                         Create
                                                                         and Test
                                                                        Data Base
                          Convert
                          Production
                            Data
                                                User
                                              Acceptance
                                                Test
                                              Data Base
                                                                         DEVELOPMENT
                             IMPLEMENTATION

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
                                    CHAPTER 8

                          TERMS AND REFERENCE MATERIALS
8.1   Glossary of Terms
      This  chapter  defines  key  terms   used  1n  OSWER's   System  Life   Cycle
Management Guidance and this practice paper.
     TERM

Access


Approval
Archive
Audit
Baseline
Change
                          DEFINITION

The  operation  of  viewing  or  copying  (including extracting)
data.

An  examination  of life cycle products,  and the results of the
project  review  process,  by  OSWER  program  management.   The
approval  process  has  three  purposes:   first, to confirm the
results   (i.e.,   the   concepts,    products,   and  management
direction)  of  life  cycle efforts to date; second, to approve
continuation  with the next stage of the  life cycle; and third,
to  confirm  the  continued  commitment  of  resources  to   the
project.   The OSWER life cycle model  requires  formal  approvals
at  the  end  of  the  Initiation  and Concept phases, and the
Definition, Design, Development, and Implementation stages.

The  third stage of the Operation phase,  and the final stage o*
the  system  life  cycle.    Its  purpose  is  to terminate the
operation of the system in an orderly, planned  manner, ensuring
that  software  and  data are properly archived or incorporated
into other systems.

A  standard-oriented  examination  of  the products and related
documentation  contained  in a baseline to assure that they are
complete,  clearly  presented,  and internally  consistent.   The
OSWER life cycle provides for five audits; Concept, Definition,
Design,  Development,  and  Operational.     Any  audit  may  be
repeated as necessary.

The  set of life cycle products and other related documentation
which  officially comprise the system at  a given point in time.
The  OSWER  life  cycle  model  provides   for  five  baselines:
Initiation,  Definition,  Design, Development,  and Operational.
The  products  contained  in  each baseline are always reviewed
prior to inclusion in the baseline.

An  alteration to the system or d.ta base(s) for maintenance or
performance  purposes,  without  affecting the functionality or
structure of the system or data base(s).
                                       44
                                                                    May 31, 1988

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Change Control
Concept
Conceptual
Data Model
Configuration
Accounting
Configuration
Management
Custodianship
Data
A  process  for  controlling modifications to  a system.   Change
control  provides  a  review  of  requested modifications,  and
consideration  of  their  impact  on  a system, before they  are
made;  it  also  ensures that changes are made in a manner that
does not disrupt ongoing system operation.

The second phase of the system life cycle.  This phase provides
a high level of functional and data requirements that relate to
an information management problem, and a comprehensive model of
a solution to meet the requirements.

A   depiction  of  data  requirements  from  an  organizational
perspective.    Corresponds to the conceptual  schema of a three
schema  -architecture  as  defined  by  the  American  National
Standards  Institute.    Entity relationship diagrams are often
used to depict the conceptual data model.

A  process  for  maintaining system baselines, including adding
products to a baseline, denoting the components of each product
(referred  to  as  configuration  items),  and  monitoring  and
recording the disposition of requested changes to the system.

A  function  which  serves to systematically identify the items
that  make  up  a  "system,  and formally control any changes or
additions  to  those  items,  in  order  to  help  maintain the
integrity of the system, and facilitate communication about the
system throughout its life cycle.

The  functions and responsibilities of an organization, such as
an  ADP  organization,  with  physical  custody  of  data   that
supports  the  work  of another organization,  such as a program
office.    For  example,  the  custodian  ensures  the physical
integrity   of   the  data  and  software  under  its  control;
safeguards  the  media storing data; ensures the data is secure
from  unauthorized  access,  change  or destruction; makes  data
accessible  to  users;  and  implements  requested  hardware or
software changes.

Representations  of facts, concepts, or instructions in symbols
suitable  for  communication,  interpretation  or processing by
human or automated means.
Data
Administration
The   management   function   responsible   for  the  planning,
definition,  organization,  protection,  and efficiency of data
and   data  bases within OSWER.  The goal of Data Administration
is   the  cost-effective provision of data of sufficient quality
to support the OSWER mission.
                                       45

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Data
Administration
Program
Data Attribute


Data Base



Data Base
Management
System (DBMS)

Data Collection


Data Definer
Data Dictionary
Data Element
Data Entity
Data Flow
A management initiative which includes policies,  standards,  and
procedures  that  increase an agency's knowledge  and  management
of  the:  composition  of  data,  source of data, processing of
data, meaning of data, flow of data,  and dissemination  of  data.
A  successful  Data  Administration  program  will  improve  the •
management of data by introducing procedures that address: data
standards,  data  requirements  determination, data definition,
data  acquisition or collection, data processing, data  storage,
data usage (including sharing and access),  and data disposal.

A  characteristic  of  a unit of data such  as length, value, or
method of representation.

A   collection   of  interrelated  data  stored  together  with
controlled   redundancy   to  serve  one or  more  systems   or
applications.

A  software system facilitating the creation and  maintenance of
a  data  base  and the execution of computer programs using  the
data base.
The   recording
organization.
and   capturing  of  data  on  behalf   of  an
The   person  or  organization  who  determines   the   essential
qualities  or  meaning  of data, and who  prescribes and  defines
procedures  which -aggregate  and  refine  data.   This includes
describing the formatting of the resulting  information to  serve
a specific decision-making context.

A  centralized  repository of information about  data,  including
its  meaning,  relationship  to  other data,  origin,  usage  and
format.    A  PIPS  data  dictionary standard  is  expected  to be
approved   late  in  1988.    Called  an  Information   Resource
Directory   System   (IRDS),   this   standard   specifies  the
capabilities   that   should   be   offered   for  future   data
dictionaries.  .  I

The  smallest  unit  of  data  that  has  meaning in describing
information.   A piece of data which would  not be meaningful if
decomposed further.

A  person,  place,  thing,  concept,  or  event   about which an
organization  may  store  data.   Data entities  are to nouns as
data  elements  are  to  adjectives.   That is to say  that data
entities  are  the  objects  being  described  by data  elements.
Entity is a synonym for data entity.

A  depiction  of  the  movement  of  information (data)  between
processes.    Data  flows are a'component of data'flow diagrams
used   by   systems  analysts  to  analyze   application  system
requirements.
                                       46

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Data
Independence
Data Integrity
Data Life Cycle
Data Security



Data Steward

Data Store


Decision Paper
Definition
Definition and
Design


Design
Development
The property of a data base management system that enables  data
to be processed independently of access mode, storage  method or
arrangement*    Data  independence  reduces  the need  to modify
application  programs  when data storage and access methods are.
modified.

The  quality  of  data  that  exists  as  long as accidental or
malicious   destruction,   alteration   or  loss  of  data   are
prevented.    This  results  in  preservation  of  data  in its
intended format, length and contents while within a data base.

The  data  life  cycle  begins  with  the definition of data to
support  new  regulations  or other program needs, and includes
strategic  data planning, data standardization, and the methods
and  standards  during  the collection, storing, accessing, and
archiving of data.

The   protection   of  data  against  unauthorized  disclosure,
transfer,  modification  or- destruction, whether accidental or
intentional.

See Stewardship.

A component of a data flow diagram that represents data at rest
or in storage.

A  decision  document  presented  to management.  It summarizes
those aspects .of-the analysis and decisions of a given phase or
stage  that  are  important to program management, and requests
approval  to  continue the project.  The OSWER life cycle model
provides  for  Decision  Papers  to  be  prepared at the end of
Concept,  Definition,  Design, Development, and Implementation.
The Mission Element Needs Statement (MENS) also is considered  a
Decision Paper.

The  first  stage  of  the  Definition  and  Design phase.   Its
purpose  is  to  define  specific, detailed functional and data
requirements  for  the  system within the context of the System
Concept.
                                             /•

The  third  phase  of  the system life cycle, consisting of  two
stages:  Definition  and Design. (See individual definitions of
each of these terms).

The  second  stage  of  the  Definition  and Design phase.   Its
purpose is to produce detailed specifications for the system to
meet the functional and data requirements within the context of
the System Concept.

The  first  stage  of the Development and Implementation phase.
Its  purpose  is  to  produce  a  system  which  is  ready   for
acceptance testing and suitable for implementation.
                                       47

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Development and
Implementation
Domain


Enhancement
Entity
Entity
Relationship
Diagram
Evaluation
Implementation
Information
Information
Flow

Initiation
The  fourth phase of the life cycle.  Its purpose is to produce
a complete system, fully tested and available for use in normal
production   mode.   There   are  two  stages  in  this  phase:
Development and Implementation.

A  set  of all values that a particular data element may posses
in actual or potential usage.

A  modification  to  a  system  that  results  in substantially
improved capabilities and, in some way, alters the structure of
the  system.    Other  modifications  that  do  not  alter  the
structure,   are   referred   to   as  changes.    Examples  of
enhancements   include  the  addition  of  new  data  elements,
changing  the  system  (or  a  part  of the system) to run in a
different   software  environment,  and  replacing  data  entry
screens   to   improve   the   user  interface  and/or  improve
performance.

A  person,  place, thing, concept, or event that is of interest
to  an enterprise.  An entity is something about which we store
data.     Examples  of  entities  are:  waste  site,  contract,
employee.   An entity has various attributes, or data elements,
which  should  be  recorded.   Examples for the entity contract
could  include  contract-number,  date, and obligation-ceiling.
Entity is a synonym for data entity.

A  diagram-  that  depicts  entities, their key attributes-(data
elements),  and the relationships between entities.  The entity
relationship  .diagram  is  the most commonly used.data modeling
technique  used  today.    Sometimes system developers refer to
entity  relationship  diagrams  as  ER  diagrams or static data
models.

The  second  stage  of  the Operation phase.  Its purpose is to
determine  whether the system is effectively.meeting the stated
requirements,  is  operating  efficiently  and  is  effectively
managed.

The  second  stage of the Development and Implementation phase.
Its  purpose is to produce a fully tested system'containing the
data  needed at start-up, and to provide needed training to the
intended users.

Any  set  of  data  which  has been aggregated by processing in
order   to   establish  a  specific  meaning  and  serve  in  a
decision-making context.

A  depiction  of the flow of information between processes.  An
information  flow  is  another name for a high-level data flow.

The  first  phase  of the system life cycle.  Its purpose is to
define  an  information  management problem within OSWER and to
determine  whether  resources  should be committed to exploring
ways to address it.

                      48

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING  THE  SYSTEM LIFE CYCLE
Initiation
Decision Paper
Life Cycle

Life Cycle
Management
Logical Data
Model
Maintenance
Metadata


Normalization
Operation
 Phase
 Privacy
A  brief  document  identifying   and  describing  an  information
management  problem.    This  document  is  prepared during the
Initiation phase.

See  'System Life Cycle11

The  process  of  managing a system through its life cycle.  As
practiced  by  OSWER,  it  is not a rigid process,  but rather a
disciplined  means  for selecting and practicing  the management
approaches and techniques that are most appropriate for a given
information management problem and/or system.

A  depiction  of  the  logical,  or programmatic,  data needed to
support an organizational mission.  The components  of a logical
data  model  include data entities and relations; data elements
or attributes; keys; secondary keys; and relationships, if data
entities  are  used.  The logical data model is a more detailed
depiction  of the conceptual data model of an organization.  It
may  correspond  to  the  external  schema  as  defined  by the
American National Standards Institute.

The  set  of  activities  that  keep  systems and data bases in
operating  condition.    Maintenance also focuses on optimizing
the  performance  of  existing  systems and data bases, without
affecting functionality or the structure of the systems or data
bases.

Data  about  data,  such  as  its  definition  or   its physical
characteristics.

The process of reducing a logical data model (structure) to its
most  basic, form,  so that the data model is stable, flexible,
and without redundancy.  A normalized data model is composed of
normalized data  entities.  A normalized data entity includes no
repeating   groups  or  data  elements  among  its  attributes,
contains attributes  (data elements) only about the  entity  being
described,  and  does not include  attributes which are dependent
upon the key of  another entity.

The  fifth  phase of the life cycle.  Its purpose is to operate
the   system   in   normal   production  mode,  monitoring  and
maintaining  its  performance, until the end of the life cycle,
and  then  to  terminate  operation.  There are three stages in
this phase: Production, Evaluation, and Archive.

The  major  segments  of the system life cycle.  There are five
phases  in  the  OSWER  system life cycle: Initiation; Concept;
Definition  and  Design;  Development  and  Implementation; and
Operation.

The  right  of   individuals  or   organizations to constrain the
collection and use  of data about  themselves.
                                       49

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
Production
Project
Project
Execution

Project
Management
Quality
Assurance
Record
      t

Review
Shared Data


Stage
The first stage of the Operation phase.  Its purpose is to make
the  system  available  to users, and make required changes and
improvements   to  ensure  that  it  contains  to  address  the
information management problem in a cost effective manner.

An organized effort to solve an information management problem.
In  most  cases,  a project extends over the entire system life
cycle.    In  some  cases  a  project  extends only through the
portion of the life cycle that can be foreseen with confidence,
e.g., through Production if the timing for ceasing operation 1s
uncertain.

The  set  of  activities which produce the concept, definition,
design, and production versions of a system.

The  set  of  activities  which  monitor  and  control  project
execution  to ensure that they are performed effectively and in
accordance  with  applicable policies, guidances and practices;
and   that   its  products  solve  the  identified  information
management need.

A function that ensures that all products of the life cycle are
substantively  accurate  and  address  the  stated  information
management  problem.  Quality assurance is accomplished through
the  efforts  or skilled professionals on the project team, and
through formal reviews.
A  group  of  related
application program.
data  elements  treated  as a unit by an
A formal examination of a life cycle product to verify that the
system   as   represented   solves  the  specified  information
management .problem.    The OSWER life cycle model provides for
five  reviews;  Concept,  Definition,  Design, Development, and
Post-  Implementation.  Any review may be repeated as necessary
to  ensure that all deficiencies have been fully and -adequately
addressed in the designated products.

Data  stored  that is created, accessed, updated, or deleted by
more than one organizational unit.

The segments of the system life cycle that occur within certain
larger  phases.    The  OSWER  system  life  cycle  divides the
Definition  and  Design  phase  into two stages: Definition and
Design.    The  development  phase  is divided into two stages:
Development and Implementation.  The Operation phase is divided
into  three  stages:  Production, Evaluation, and Archive.  The
phases Initiation and Concept are not divided into stages.
                                       50

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE  SYSTEM LIFE CYCLE
Stewardship
System
System Concept
System Life
Cycle
Threshold
Analysis

Walkthrough
The  functions and responsibilities of an organizational  entity
that   exercises   control   over  data  on  behalf   of  OSWER.
Organizations  that  require  data  to be collected,  processed,
stored  or  used in support of OSWER1s mission  have  stewardship
responsibilities.    These  responsibilities  include  ensuring
that:  (1) Only data relevant to OSWER's missions  is  collected.
(2)  Data that is collected is of sufficient  quality  to support
OSWER's  missions.    (3)  Data  is reused wherever  appropriate
within  OSWER.    (4) Data is clearly defined and  documented in
compliance  with established directives.  (5) Systems practices
under  the  organization's  stewardship  conform  to  EPA  Data
Administration guidance.

An  organized  set  of  functions,  data, procedures, hardware,
software,  communications,  and/or  documentation  which enables
OSWER  to  solve  a specific information management  problem.  A
system need not necessarily be automated; but most instances of
life  cycle  management  will  apply  to  automated  information
systems.

A  high-level complete description of a system  (including data,
processing     capabilities,     hardware,     software     and
communications).    It is produced during the Concept phase and
serves  as both a check on the validity and completeness of the
problem,  and  the  basts for defining more detailed functional
and data requirements.

The evolution of a system from the 'initial identification of an
information  management  problem  through system termination or
replacement.

The  process of determining the appropriate review and approval
levels for an OSWER system project.

A  highly-structured  meeting  to  review  the  completeness and
quality  of  selected module(s) of the system,  or  of the entire
system.    Walkthroughs  are  usually  conducted by  the project
team,  often  are  attended by user representatives, and may be
held    at    any    point    1n   the   system   Hfe   cycle.
                                       51

-------
         PRACTICE PAPER FOR DATA MANAGEMENT DURING THE SYSTEM LIFE CYCLE
8.2 Additional Reference Materials
8.2.1  Data Modeling References
    o  Entity Modeling. Ronald Ross, Database Research Group, Boston, 1987.
    o  Guide on Logical Database Design. Elizabeth Fong, et al., National  Bureau
       of.Standards Publication 500-122, 1985.
8.2.2  Physical Data Base Design References
    o  Computer  Data  Base Organization. James Martin, Prentice-Hall,  Englewood
       Cliffs, New Jersey, 1975.
    o  Data   Base;	Structured  Techniques  for  Design,  Performance,   and
       Management, S. Atre, A. Wiley Series, New York, 1980. ^
    o  Design  and  Strategy  For  Distributed' Data  Processing,  James Martin,
       Prentice-Hall, Englewood Cliffs, New Jersey, 1981.

8.2.3  Data Documentation References
    o  Data Dictionary/Pirectory Systems, BeIkis Leong-Hong and Bernard Plagman,
       John Wiley & Sons, New York, 1982.    •
    o  Data  Dictionaries  and  Data  Administration.  Ronald  Ross, AMACOM,  New
       York, 1981.
    o  Guide  on  Data Entity Naming Conventions, Judith Newton, National  Bureau
       of Standards Publication 500-149, 1987.             .      '
                                        52

-------
                      OSWER DIRECT We Wi^cc.j*.^
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
    Part 3: Practice Paper
  Project Management Plan
       January, 1989

-------
                        TABLE OF CONTENTS


                                                            Page

1.   Practice Paper Purpose	   1


2.   Structure and Contents of Project Management  Plan   	   2

    2.1.   Project Charter/Objectives	   2

    2.2.   Life Cycle Adjustment	   2

    2.3.   Project Team Oraanization  .:	   4

    2.4.   Project Budget  	   6

    2.5.   Project Reviews/Quality Assurance  	  7

    2.6.   Applicable Project Approvals  	  8

    2.7.   Benefit-Cost Analysis  	  9

    2.8.   Methodologies.and Tools   	  9

    2.9.   Workplan  	,		 11

    2.10. Procurement Approach	 12

    2.11. Configuration Management Plan  	'.	 13

    2/12. Documentation Standards	 14

    2.13. Security Approach   	 14

    2.14. Conversion Approach	 15

    2.15. Installation Approach  	»	 16

    2.16. User Support Approach  	 16

    2.17. Maintenance Approach	 17

    2.18. Operations Approach   	 18


3.   Development and Update of the Project Management
    Plan	 19

    3.1.   Responsibility for Developing and Updating the
          Project Management Plan   	 19

-------
                                                      ?»v—n L.':nlw'«V— >. •-.,—••—--
                   TABLE OF CONTENTS (Continued)


                                                             Page

  *   3.2   Format of the Project Management Plan  	 19

     3.3   Evolution of the Project Management Plan Through
           the system Life Cycle	 19

     3.4   Retention of Old Project Management Plans  	 22
 4.   Relationships Between Project Management Plan Topics  .. 23
                             EXHIBITS
                     ~s


•2-1.   Outline of Project Management .Plan  	  3

 3-1.   Evolution of Project Management Plan Through the
       System Life Cycle	 20

 4-1.   Summary of Relationships Between Project Management
       Plan Topics	»	 24

 4-2.   Details of Relationships Between Project Management
       Plan Topics 	.	 25


 Appendices

 A.  Detailed Outline of Project Management Plan   	   34

-------
                                                      OSWER DIRECTIVE #S023.GOa
                    1.  PRACTICE PAPER PORPOSE


    This  practice  paper  constitutes a section of Part 3 of the
Office  of Solid Waste  and Emergency Response (OSWER)  System Life
Cycle  Management  Guidance.   It describes the Project Management
Planr  a  key  document  of  the system life cycle.  Every system
project is required to  develop and use a Project Management Plan.
This  practice  paper  describes the structure and content of the
Project  Management  Plan,  and  its evolution through the system
life cycle.

    OSWER  places  great  emphasis on the Project Management Plan
because  of the intrinsic difficulty of managing system projects,
especially  projects for systems that support more than a handful
of  individuals.    Rigorous  development  and use of the Project
Management  Plan will help ensure that important issues regarding
the  approach  to  the   project  are carefully considered and the
decisions are documented.  The Project Management Plan also helps
to  communicate  the  approach and coordinate the approach across
all  project  team  members,  and  to clearly measure progress in
completing the project.

    The topics addressed in this practice paper include:

    o  The structure and content of a complete Project Management
       Plan;

    o  Responsibility  for.  preparing  and  updating  the Project
       Management Plan;

    -o  How the Project Management Plan evolves through the system
       life cycle; and                               ,

    o  How  the  components of the Project Management Plan relate
       to each other and to the other products of  the system life
       cycle.

    The Project Management Plan serves several important purposes
in support of the system life cycle:

    o  Helps   ensure  that  important   issues  are  purposefully
       considered and that key decisions  are  clearly documented;

    o  Helps  support  the  coordination  of various organizations
       and individuals involved in a  system project, by providing
       a single known source of information regarding the project
       approach and the role of each  organization/individual; and

    o  Provides a basis for measuring progress through  the system
       life cycle, and the potential  impact of any changes to the
      'approach for'conducting the project.

-------
    This  practice  paper  is not intended as a primary means for
training  project managers.  Rather, it describes an approach for
clearly   documenting   certain  project  management  topics  and
decisions  of  importance to OSWER.  It should also be noted that
although   the   Project   Management   Plan  references  certain
characteristics  of  an information system (e.g., software tools,
security),  the  Project  Management Plan describes the logistics
for  the  project.    It  does  not serve as documentation of the
system  requirements,  design  or  other  features  of the system
included in other formal system documentation.
      2. STRUCTURE AND CONTENTS OF PROJECT MANAGEMENT PLAN


      The  Project  Management  Plan  should  use  the same basic
structure  for all systems projects.  This structure is presented
in  Exhibit  2-1.    All  topics  shown  in Exhibit 2-1 should be
included  in the plan; however, the level of detail at which each
is discussed should be tailored to the individual project.

      No  specific  format  is required for most topics; however,
certain specific information should be provided.  This section of.
the  practice paper, identifies the information to be provided for
each  topic of the Project Management Plan, and suggests specific
formats or presentation techniques where appropriate.  Appendix A
provides   a  more  detailed  outline  of  the  complete  Project
Management Plan.

2.1.  Project Charter/Objectives

      Every   system   project   should  have  a  clear  charter,
describing  the  objectives  of the project and certain other key
project  attributes.  This section of the Project Management Plan
provides  the overall context for the other sections of the Plan.
It   summarizes   the  following  information  from  the  Project
Initiation Decision Paper:               :

      o  The information management problem to be solved,
                                                   t
      o  The  scope of the problem in terms of OSWER programs and
         organizations,

      o  The timeframe for solving the problem, and

      o  The  organization(s)  and  individual(s)  that  serve as
         programmatic sponsor for the project.

2.2.  Life Cycle Adjustment

      Parts 1 and 2 of this Guidance describes specific sequence
of  life ' cycle phases and stages, a sequence  that applies  to the
entire  system.  For some projects, it may be  desirable to  adjust

-------
                                                    UN. I I -  - ' • -"
EXHIBIT 2-1: OUTLINE OF PROJECT MANAGEMENT PLAN
                          TOPICS

              Project Charter/Objectives
              Life Cycle Adjustment
              Project Team Organization
              Project Budget
              Project Reviews/Quality Assurance
              Applicable Project Approvals
              Benefit-Cost Analysis
              Methodologies and Tools
              Workplan
              Procurement Approach
              Configuration Management Plan
              Documentation Standards
              Security Approach
              Conversion Approach
              Installation Approach
              User Support Approach
              Maintenance Approach.
               Operations Approach

-------
the  life  cycle,  such  as  by  combining  certain stages, or by
dividing  the  system  into  different modules, each with its own
schedule for progressing through the life cycle.  This section of
the  Project  Management  Plan is extremely important, because it
establishes  the  framework for many other sections, particularly
the  project  Workplan.    This section describes any significant
planned   adjustments  to  the  conventional  system  life  cycle
described  in Parts 1 and 2 of this Guidance, and the reasons for
such  adjustments.    Examples  of  the types of adjustments that
should be included in this section are:
     »»-
      o  The  consolidation  of  portions  of two or more stages,
         such   as  the  generation  of  software  (part  of  the
         Development stage) during the Design stage,

      o  Partitioning  the project or system into modules or work
         packages  (usually  done during the Concept phase), with
         different life cycle schedules for one or more modules,

      o  Phased  development  of  the  system  or data base using
         multiple  life  cycles  —  one  to provide basic system
         capabilities,  and subsequent cycles to provide expanded
         capabilities  through  the  planned replacement of major
         portions of the system,

      o  Iterative cycling through portions of the life cycle, as
         is 'often  the  case  in  the  development  of an expert
         system,

      o  Consolidation of two or more system life cycle products,
         including consolidation of System Decision Papers, and

      o  Elimination of any system life cycle products.

2.3.  Project Team Organization

      This  section  describes  how  the  project  team  will  be
organized  in terms of the specific organizations and individuals
who  will  participate  actively in the project.  This section is
particularly  useful  for large projects, with many participating
organizations  and individuals.  The Project Manager may use this
section of the Plan as a stand-alone document, distributing it to
all   participating  organizations  (including  contractors)  and
individuals to improve project coordination.

      Specific  information  contained  in this section should be
documented   using  an  organization  chart,  as  well  as  other
applicable techniques, and includes:

      o  Identification  of  the Project Manager, his/her  current
         home organization, and any assignment of this  individual
         to  another  organization  (e.g.,  detailing   to  another
         office) to accomplish his/her role as project manager;


-------
                                               OSWEP DIRECTIVE *?3023.DOa
o  Identification of any supporting organization structures
   that  will  serve  in a project management role, such as
   boards  and  advisory  committees,  and  the  roles  and
   authorities  of such organizations.  These organizations
   may  be  unique  to  the  system,  or  may  be  standing
   organizations   with   management  responsibilities  for
   systems affecting a designated program;

o  Description   of   project  staffing,  including  Agency
   personnel and contractor support.  The home organization
   and  percentage  and duration of assignment and for each
   Agency   team   member  should  be  clearly  identified.
   Specific  contractor  organizations should be identified
   as  soon  as  is  practical.    Total  contractor  staff
   assigned  to  the  project, and key contractor personnel
   should  be identified as well.  The roles of each member
   of  the  project  team should be clearly identified on a
   person-by-person  basis  or, for very large projects, by
   identifying  the  specific sub-team to which each member
   is  assigned.    Experts  in  programmatic  or technical
   subject  matter  of particular importance to the project
   should be clearly identified.

o  Description  of  the  structure  of  the  project  staff.
   reporting   to   the   project  manager,  including  the
   identification of any sub-teams  (if applicable) and size
   and team leaders for each team;

o  Identification  of  the data steward for the project, or
   multiple stewards if appropriate,  for different types of
   data;

o  Identification  of individual organizations  that have.an
   interest  in the system and are  not directly represented
   on the project team, but which will be informed of major
   milestones  and  decisions  through  the distribution of
   required  system  decision papers  and other materials as
   appropriate.  Examples include:

   .—.  Individual   regional   waste   management   program
       organizations,

   —  Office of Information Resources Management,

   —  National Data Processing Division  - NCC  and WIC,

       Individual .regional ADP organizations,

   —  Individual    State    waste   management   program
       organizations, and

   —  Office of the Inspector General;

-------
                                                       OSWER DIRECTIVE #332E.W
         Identification  of   the  members  of  the Change Control
         Board,   and   the   authority  of  the  Board  (i.e.,  a
         decision-making  body or an advisory body to the Project
         Manager).
      OSWER  requires  the  use  of  block  diagrams,  or similar
techniques,   to   illustrate   the  project  team  organization.
Multiple  diagrams  should  be used to illustrate team structures
that are expected to change throughout the life cycle.

      A  separate  System Life Cycle Management Guidance Practice
Paper  entitled 'Project Participation and Coordination1 provides
suggestions   for   identifying   the  organizations  who  should
participate  in  each  system  project.  Of vital importance, the
contents  of  the  Project  Management Plan should be coordinated
with  all the organizations that will be involved in the project.
The level of commitment of Agency staff to the project, and their
commitments  to  other  assignments,  must  be  agreed  on by the
Project Manager and each participant's supervisor.

2.4.  Project Budget

      This  section  identifies the approved resources to be used
to  accomplish  the  project,  the  source  of  funding  for  all
resources  in  terms  of organizational entities (e.g., allowance
holders  and ^suballowances),  and  the  accounting  methods  and
procedures  that will be used to monitor the project budget.  The
Project   Budget  section  of  the  Project . Management  Plan  is
particularly  important  because  it  describes  a  commitment of
resources, and not just a need for resources.  The project budget
is  broken  out  for  each  phase  and  stage, and identifies the
resource  level  and cost of the following types of .resources, as
applicable:

      o  EPA staff,

      o  Contractor services,

      o  Equipment purchase or lease,

      o  Equipment maintenance,                     /

      o  Site preparation (e.g., to accommodate ADP equipment),

      o  Software package(s) purchase or lease,

      o  Supplies,
                      i
      o  Computer  timeshare  (internal to EPA such as ase of the
         National   Computer   Center   mainframe,• and  external
         services)

      o  Other costs

-------
                                                      OSWEn D!r£C::;'.£ i;Z~^.-^.
      For  some  projects,   the  Project  Budget  may also serve to
indicate  the  need  for  additional  resources  — the difference
between  the resources needed for each phase and/or stage and the
commitments received to date.

      OSWER  places  particular  emphasis on effective monitoring
and  control  of  project resources and budgets.  For each of the
above  resources,  this  section  of  the Project Management Plan
identifies  the  procedures  and  tools  to  be  used to track the
expenditure  of  project resources against the budget provided by
each funding source.

      Of  particular  note, the Project Budget (together with the
Benefit-Cost  Analysis)  serves as the source of cost information
used  to  determine  the appropriate level of review and approval
for the project  (i.e., Threshold Analysis).

2.5.  Project Reviews/Quality Assurance

      This  section  identifies  the  individual  formal  project
reviews  and  other  quality assurance activities to be conducted
during  the system life cycle.  Project reviews are a key step in
each  phase  and stage of the life cycle — they provide feedback
to  the  project  team,  and are advisory to the project approval
authority  who  will  be asked to approve the continuation of the
project.    (The  required reviews, and technique for determining
who should conduct them (i.e., Threshold Analysis), are described
in   the  practice  paper  on  'System  Life  Cycle  Reviews  and
Approvals'.)

      Some  of   the  information contained in this section of the
Project  Management  Plan  will be developed by the Lead Reviewer
for  the  project, and should be provided to the Project- Manager.
Specific information contained in this section includes:

      o  Identification   of   the   applicable   'threshold',  or
         organizational  level  for  conducting required reviews.
         For  a Level I system, also designates the criteria  that
         result  in the Level I classification;
                                                    /
      o  Identification of the specific formal project  reviews to
         be  conducted  in  each phase and stage,  and approximate
         schedule.   The number of reviews and schedule should be
         structured to reflect any adjustments  to  the system  life
         cycle.

      o  Identification   of   the   specific  organizations  and
         individuals   who   will  participate  in  each   review;
         designated  individuals  should  be  independent  of the
         project team;

      o  Description  of how the reviews are to be conducted, and
         the  approach/procedure  to  be  used  to  document  the
         results of reviews; and

-------
      o  Drawing  from  the  project  Workplan, identification of
         other   activities   to  be  conducted  to  confirm  the
         programmatic  and technical findings and recommendations
         of  the  project  team (e.g., system design walkthroughs
         and presentations, circulation of life cycle products to
         user  and  other  organizations for comment, independent
         validation and verification (IV&V), acceptance testing).
         To  ensure  that the project team will effectively solve
         the   information   management   problem,   these  other
         activities  are strongly encouraged.  Reviews should not
         be  limited to only the formal reviews and specified for
         each phase of the life cycle.

2.6.  Applicable Project Approvals

      This  section  identifies  the  individual  formal  project
approvals  to  be  obtained  during the system life cycle.  OSWER
requires  that every project be approved at the end of each phase
and  stage  of  its  life  cycle to ensure that it will solve the
information  management  problem, within an acceptable timeframe,
and  with  reasonable  resources.    (The required approvals, and
technique for determining the approval authority (i.e., Threshold
Analysis),  are  described  in the practice paper on 'System Life
Cycle Reviews and Approvals'.)  Specific information contained in.
this section includes:

      o  Identification   of   the   applicable  'threshold',  or
         organizational   level   for   providing   the  required
         approvals.    For  a Level I system, also designates the
         criteria that result in the Level I classification;

      o  Identification  of the specific approvals to be obtained
         in  each phase and stage, and approximate schedule.  The
         points  of  approval  and  approval  schedule  should be
         structured to reflect any adjustments to the system life
         cycle.

      o  Identification   of   the   specific  organizations  and
         individuals   who   will  participate  in  the  approval
         process,  and  the  means  to  be used to present system
         decision  papers  and  other  life  cycle 'products  (as
         appropriate) to the approval authority;

      o  Description  of  the  approach/procedure  to  be used to
         document the results of each requested approval;

      o  Drawing  from  the  project  Workplan, identification of
         other  approvals  tc  be  secured  by  the  project,  in
         addition  to  those  identified  in  Part 2 of the OSWER
         System Life Cycle Management Guidance;

-------
                                                     OSWER DIRESTtVS ^222-
2.7.  Benefit-Cost Analysis

      This  section provides a summary of the system benefit-cost
analysis.      This analysis is first presented in the Initiation
Decision Paper as an initial rough estimate of project scale, and
a  comprehensiver  detailed  analysis  is  conducted  during  the
Concept  phase  and  is contained in the System Concept document.
The  Benefit-cost  analysis  presented'  in  this  section  of the
Project  Management  Plan  draws on these life cycle products for
both benefit and cost information.  As the system evolves through
the  life  cycle,  this  analysis  must  be updated.  The current
perspective  of  benefits  and costs is documented in detail as a
refinement  to  the  System  Concept (contained in the Initiation
Baseline)  and  is  documented in summary form in this section of
the  Project  Management Plan.  Specific information contained in
this section includes:

      o  Analytic  methodology  and  major  assumptions regarding
         program  direction,  information  management technology,
         resource    availability,   and/or   other   issues   as
         applicable;

      o  System benefits:

         —  Program   effectiveness    (quantified   as  specific
             measures of improvement if possible),

         —  One time monetary benefits,

         —  Recurring/annual monetary benefits;

      o  System costs:

         —  Initial  investment   (e.g., Initiation phase through
             Implementation stage),

         —  Recurring/annual costs,

         —  Total system  life cycle costs;

      o  System payback period; and

      o  Sensitivity  • of   estimated   benefits   and  costs   to
         identified assumptions.

      Of particular note,  the costs documented  in this section  of
the  Project  Management   Plan, together with the Project Budget,
serve  as  the  cost  information  needed to  conduct the Threshold
Analysis for project reviews and approvals.

2.8.  Methodologies and Tools

      This  section  provides  a summary of  the methods and tools
selected  to  conduct  the activities  of the  system life  cycle.

-------
Each  phase and stage of the life cycle should be conducted using
an  appropriate  set  of systems analysis and development methods
tools.    This  section of the Project Management Plan identifies
the  methods  and  tools  to  be used, and also describes how the
methods  and tools will work together.  It also describes how the
tools  will  be  used  to  produce the required documentation and
other  products  of  the  life  cycle, and any adjustments to the
products (per the outlines contained in Part 2 of this Guidance).

      This  section  draws  from  the  System Concept the initial
selections  of methods and tools.  These selections are confirmed
during  subsequent  phases  and  stages,  and  any new or changed
selections  are documented in summary form in this section of the
Project  Management Plan.  Examples of the types of methodologies
and  tools  identified  in this section of the Project Management
Plan include:

      o  Techniques   and   software   tools  to  support  system
         requirements analysis (e.g., system prototyping),

      o  System  analysis and design methodologies (e.g., Yourdon
         structured analysis, application generators),

      o  Techniques  for data analysis (e.g., entity-relationship.
         analysis),

      o  Computer-aided software engineering (CASE) tools,

      6  Programming languages (e.g., COBOL)

      o  Programming  aids  and  debugging tools (e.g., OPTIMIZER
         III),

      o  Communications software (e.g., CICS, Kermit),

      o  Data base management software (e.g., ADABAS),

      o  File  management  and  configuration management software
         tools (e.g., TIP Repository),

      o  Project management tools (e.g., TELLAPLAN,'SUPERPROJECT)
         and;

      o  Word processing software (e.g., WORDPERFECT).

      Specific   information   contained   in  this  section  for
individual selections of methodologies and tools includes:

      o  Identification of methodology or tool,
      o  Training/other special support required, and
      o  Procurements needed for acquisition and/or support.
                               10

-------
                                                     OSWER DIRECTIVE #£023.COa


      As  illustrated  in  Exhibit  3-1,  the selections for  each
phase  and  stage  are  finalized  at  the end of the immediately
preceding phase/stage.

2.9.  Workplan

      This   section   describes  in  detail  the  logistic;;   for
conducting  the  project.    It  is  structured  to  parallel the  .
individual  phases  and  stages  of  the  system life cycle.   The
workplan describes the specific tasks for conducting the project,
noting  the  relationships  between tasks.  For projects that are
very large, complex, or on a very tight schedule, the workplan is
particularly  important  —  it identifies the 'critical path1 of
activities  that  are instrumental to the success of the project.
The  workplan also identifies resources for each task, serving to
clearly allocate the resources provided in the Project Budget.

      The  Workplan is most detailed for the immediately upcoming
phases  or stages and, as illustrated, in Exhibit 3-1, is examined
in  detail  and  confirmed  for  each  phase  or  stage  prior to
initiation of work in that phase or stage.  The Workplan contains
the following information for each phase/stage:

      o  Identification  of  all  project  activities,  and  work-
         breakdown  of  activities  into  more  discrete tasks as
         appropriate;

      o  Identification   of   all .  products,   and  mapping  of
         activities/tasks to products;

      o  A  schedule   (i.e.,  start and completion dates for each
         activity/task)  documented  in  the  format  of  a Gantt
         chart,   including  the  schedule  for  required  formal
         reviews and approvals *;

      o  Agency   staff   and   contractor  assignments  to  each
         activity/task *;

      o  Level of resources/funding for each activity/task and/or
         -life cycle product *;

      o  Schedule  relationships/dependencies  between  activities
         and/or  tasks,  including  dependencies  with   regard  to
         activities/tasks for other phases or stages  *;  and

      o  For  very  large  projects,  where the  system  is divided
    • .    into  modules  or  work  packages, it will  be  helpful  to
         prepare  a  high  level workplan integrating the project
         tasks  across  modules, and a more detailed workplan  for
         each module.  *
•
      o  For   projects  involving  a  procurement   of   hardware,
         software,  or support services, one part of  the Workplan


                               11

-------
                                                      OSWER DIRECTIVE #9028.00a


         should  be devoted to the activities of the "Procurement
         Approach" for the project.

      For  those  items denoted above with an asterisk {*),  it is
suggested  that  automated  project  management  tools be used to
develop  and  document  the corresponding portions of the project
Workplan.    The project Workplan also identifies the approach to
be  used  for  project  status  reporting,  including procedures,
report  content,  frequency,  and  assignments  of  personnel  to
perform status reporting.

2.10. Procurement Approach

      This section summarizes the means to be used to acquire all
contract  support  services,  to  acquire  any  needed  hardware,
software,  and communications capabilities that are not currently
installed  at  needed locations, and to obtain support from other
government  organizations  (e.g.,  interagency agreements).   Most
projects  include  at least one significant resource acquisition,
and  the  Procurement  Approach  helps  ensure  that  the  needed
resources  can be obtained and are available at the time they are
needed.

      The  Procurement Approach should be complete for all stages .
through  Production  by the end of the Concept phase if possible,
and no later than the end of the Definition stage, to ensure that
adequate  lead time is available to acquire needed resources.  It
is  important  to  prepare this section of the Project Management
Plan  even if the project intends to acquire resources through an
existing  contract.    Specific  information  contained  in  this
section includes:

      o  Resources   to   be   acquired  through  existing  OSWER
         contracts:

         —  Resource  identification  (e.g.,  specific hardware,
             software, communications or service),

         —  Contract identification,
        •                              »             /
         —  Planned acquisition date, and

         —  Lead contact person on project team;

      o  Resources  to  be acquired through existing contracts of
         other Agency offices:

         —  Resource identification

         —  Contract identification,

         —  Planned acquisition date, and

         —  Lead contact person on project team;

                                12

-------
                                                      CSWER DIRECTIVE #902S.OOa
      o  Resources  to be  acquired  through  new procurements:

         —   Resource identification/

         —   Planned procurement award date,

             Scope   of procurement  anticipated  (procurement  for
             single  project/system  or a procurement  to  support
             multiple projects/systems),

         —   Type   of   procurement   (e.g.,    full    and  open
             competition,   limited   competition,    sole   source
             award),

             Lead contact person on project team,

         —   Lead  contact  person(s)  at   other  Agency organiza-
             tion(s) providing procurement assistance,  and

             [Note __  that   the   wofkplan   (tasks,   milestones,
             schedules,    staffing)    for   accomplishing   all
             activities  needed  to  complete  the procurement is
             included  in  the  "Workplan" section of the Project
             Management Plan;

      o  Support  to  be  acquired  from  other  EPA  offices and
         government   organizations    (e.g.,   General   Services
         Administration):

         —  Organization identification,

         —  Type of support needed,

         —  Planned start date for support,

         —  Lead contact person on project team, and

         —  Lead contact person at support'organization;
                                                       >
         -—-  Tasks and task schedule for establishing needed
             agreement.

      Some  of  the  content  of  the  Procurement Approach may be
sensitive,  and  should  be  maintained and stored in a manner to
prevent  disclosure t to  contractors in advance of the proper  time
for formal notification  of upcoming procurement "actions.
                                          i
2.11. Configuration Management Plan

      This  section  describes  the  organization, procedures and
tools  used to identify, monitor and control the configuration of
the  system.    Configuration Management  is an important  function
within  the  OSWER  system  life cycle —  it serves to ensure the
integrity   of  the  system  throughout   its  life  cycle.    The

                               13

-------
Configuration Management Plan describes in detail how the project
will  conduct  Configuration  Management.    Specific information
contained in this section includes:

      o  Identification of Configuration Manager,

      o  Identification  of Change Control Board — organizations
         represented and individual members, and authority of the
         Board.

      o  System  baselines (identification/index of configuration
         items),

      o  Change request review and approval procedures,

      o  Configuration status accounting procedures, and

      o  Software control procedures.

      Some  Configuration Management "Plans may be quite  long, and
can be maintained as a stand-alone document that is referenced in
the Project Management Plan.

      A  separate  System . Life  Cycle  Management Practice Paper.
entitled  'System  Configuration  Management1   describes  OSWER's
practice of configuration management, and explains in more detail
the content of the Configuration Management Plan.

2.12. Documentation Standards

      This  section  identifies  the  standards  to  be   used  in
producing  the  system  documentation  required in each  phase and
stage  of  the  system  life  cycle.   Standards are particularly
important    when   contractor   staff   are   preparing   system
documentation,  because  these  standards  are  a  key  basis for
determining   whether   the  contractor  has  delivered   adequate
documentation.

      This  section includes the identification of specific OSWER
standards, . standards  prescribed  by  the  Agency, .and  standards
adopted from other organizations, such as the Federal Information
Processing  Standards  (FIPS)  issued by the Bureau of Standards,
Department  of  Commerce.  In the absence of a mandatory standard
for  a  system life cycle product, this section should identify a
comparable  product(s) produced by other projects that will serve
as a model for the current project.
    f
2.13. Security Approach

      This   section   provides   a   summary   of  the  security
requirements and security features of. the system.  It is included
in the Project Management Plan to provide an overview of security
needed  to  prepare  and  review  the  Project Workplan and other
sections of the Project Management Plan.  This section draws from

                               14

-------
                                                       OSWER DIRECTIVE *302a.QCa
the  System  Concept,   Detailed Functional Requirements,  Detailed
Data  Requirements, and System Design to provide a summary of the
system  and  data  security  requirements and the system features
that  meet these requirements.  Specific information presented in
this section at a summary level includes:

      o  Functional security requirements,

      o  Data  security requirements, including identification of
         confidential or sensitive information,

      o  Project   team   organization  to  develop  and  support
         specific   security   features   and   capabilities  (if
         applicable),

      o  Hardware and facilities access security measures,*

      o  Software and communications security measures,

      o  Data base security measures,'

      o  Procedural   measures  (e.g.,  procedures  for  handling
         confidential  or  sensitive  input  documents and system
         outputs), and

      o  Software and data base, backup and recovery measures.

2*14. Conversion Approach

      This  section  draws  from  the  System   Concept,  Detailed
Fxmctional  Requirements,  Detailed Data  Requirements, and System
Design  to  provide  a  summary  of the data to be converted from
existing  systems  and  data  bases,  conversion  activities and
procedures,  and  organizations responsible for accomplishing the
conversion.    It  is  included in the Project  Management Plan  to
provide • an overview of the conversion approach needed to prepare
and review the Project Workplan and other sections of the Project
Management  Plan.  Specific information presented in this section
at a summary level includes:
                                                    y
      o  Identification  of   major types  of data  to be converted,
         including  conversion  of  data   currently maintained  in
         hardcopy  form  using  manual  procedures  as well as the
         conversion  of  data currently   maintained by  automated
         systems;

      o  Identification of the following  for each type of data  to
         be converted:

         —  Source and location of data,                       .

         —  Anticipated data quality problems,
                                15

-------
                                                    CSWER DIRECTIVE


         —  Organizations) responsible for data cleanup,

         —  Organizations)   responsible   for   planning   and
             conversion,

         —  Conversion schedule, and

         —  Reference    to    specific   sections   of   system
             documentation    describing    detailed   conversion
             procedures and software.


2.15. Installation Approach

      This  section  draws  from  the  System  Concept and System
Design  to  provide a summary of the logistics for installing the
system  and  data  base  in  .the  production  environment.  It is
included in the Project Management Plan to provide an  overview of
the  installation  approach  needed  to  prepare  and   review the
Project  Workplan  and  other  sections of the Project Management
Plan.    Specific  information  presented  in  this section at a
summary level includes:

      o-  Identification of the major modules/components that will-
         be separately installed items;

      o  Identification  .of  the  facilities  and  location(s) at
         which  the  system  and data base will be installed, and
         the  specific modules/components to be installed at each
         location;

      o  Identification  for  each  module/component installed at
         each location:

         —  Installation date,
         —-  Special conditions (if any),
         —  Organizations  and specific personnel to  perform the
             installation, and
         --  Organizations  and  specific  personnel  on  call to
         •  .  support the installation;

      o  Mechanisms  to ensure effective software integration and
         data bases synchronization for system modules/components
         installed at multiple locations.

2.16. User Support Approach

      This  section  draws  from  the  System  Concept,  Detailed
Functional  Requirements,  and System Design to provide a summary
of  the  activities  and  materials to be used to conduct initial
system training and provide ongoing user support.  It is  included
in the Project Management Plan to provide an overview of  the  user
support  approach  needed  to  prepare  and  review  the  Project
Workplan  and  other  sections  of  the  Project Management Plan.

                               16

-------
                                                      OSWER DIRECTIVE #9Q2B.03a
Specific information presented in this section at a summary level
includes:

      o  Lead  organizations for planning and conducting training
         and ongoing user support;

      o  Identification  of  individual  training  sessions to be
         conducted  in  support of initial system implementation,
         and for each session:

         —  Location, date and time,

         —  Intended  trainees  and subject material  (e.g., data
             entry/edit/update    procedures,    reporting    and
             retrieval, system administration, etc.)r
                                                   «
         —  Session  format  (group  presentation/demonstration,
             one-on-one training), and

         —  Organizations   and  individuals  who  will  conduct
             training;

      o  Identification  of  other training activities/materials,
         such  as tutorials, computer-based training, etc.

      o  Identification   of   user  support   functions   such   as
         hotlines,   user   groups,   etc..,  including for  each
         function:

         —  Function identification,
         —  Expected duration,
         —  Staffing level,
         —  Assignments    of    specific     organizations    and
             individuals, and
         —  Physical location(s).
  •
•2.17. Maintenance  Approach

      This   section draws  from  the   System  Concept,   Detailed
Functional •  Requirements,   and System Design  to  provide a summary
of   the  organizational  approach  for   maintaining  the   system.
System   maintenance is  crucial  to  the ongoing viability of  the
system.      For    distributed  systems,  system  maintenance   is
particularly  challenging,  and the Maintenance  Approach  takes on
added  importance.    This  section   is  included  in  the Project
Management   Plan   to  provide  an  overview  of   the   Maintenance
Approach needed   to  prepare and review the Project Workplan  and
other    sections   of  the   Project  Management  Plan.     Specific
information  presented in this section  includes:

      o  Identification     of   organizations    responsible   for
         performing software maintenance for  each system  module;
                                17

-------
                                                        33 DP.ECTIVE ssrsa.oo
      o  Identification    of   organizations   responsible   for
         maintaining  applications  software  packages, including
         any customized portions of the package;

      o  Identification    of   organizations   responsible   for
         maintaining  each interface with other automated systems
         and data bases;       .
      o  Identification    of   organizations   responsible   for
         supporting   the   release   of  new  software  (routine
         maintenance and enhancements) at each location where the
      "  system is installed; and

      o  Identification  of  currently  planned  maintenance  and
         system   enhancement   releases,  and  the  content  and
         installation schedule for each release.

      Other  documents  provide  additional, detailed information
regarding  system  maintenance: details of software libraries and
maintenance  procedures are documented in the Maintenance Manual,
and  are summarized in the Configuration Management Plan (another
section  of the Project Management Plan).  These documents should
be specifically referenced by the Maintenance Plan.

2.18. Operations Approach

      This  section  draws  from  the  System  Concept,  Detailed
Functional  Requirements,  and System Design to provide a summary
of  the  organizational approach for operating the system.  It is
included  in  the Project Management Plan to help ensure that all
organizations with system operations responsibilities are clearly
designated  and are informed of their responsibilities.  Specific
information presented in this section includes:

      o  Identification    of   organizations   responsible   for
         performing   basic  system  operations  —  data  entry,
         update, and reporting — for each module of the system;

         Identification    of   organizations   responsible   for
         performing  system and data base backup and recovery for
         each   facility  (including  individual  microcomputers)
         where the system is installed;
each   facility  (including  i
where the system is installed;
      o  Identification    of   organizations   responsible   for
         performing  other system administration functions (e.g.,
         maintenance  of data tables) for each facility where the
         system is installed; and

      o  Reference  to the Data Management Plan for the system to
         identify   organizations   responsible  for  other  data
         administration functions.

      Other  documents  provide  additional, detailed information
regarding  system operations: details of operating procedures are


                               18

-------
documented  in  the  Operation Manual and User Manual.   Organiza-
tions  responsible  for  providing technical  support  to  users  are
identified  in  the  User  Support  Plan  (another  section of  the
Project Management Plan).  These documents should be  specifically
referenced .by the Operations Plan.
      3. DEVELOPMENT AND UPDATE OF THE PROJECT MANAGEMENT PLAN

3.1.  Responsibility for Developing and Updating the Project
      Management Plan

      The  Project  Manager  is  responsible  for  developing the
Project Management Plan and for keeping it current throughout the
system life cycle.  An out of date Project Management Plan is not
useful  to  guide  the project, and could lead to confusion among
project  participants.    The  Project Manager may be assisted by
other individuals as appropriate.

3.2.  Format of the Project Management Plan

      All  sections of the Project Management Plan should be kept
in  a  single  document,  organized  in accordance with the major.
topics  of  the  Plan.  . Use of a three-ring binder or .binders is
recommended.    For those portions of the Project Management Plan
developed  and  maintained  using .automated  project  management
tools,  current  outputs  of  the tools should be included in the
binder, if possible.  Certain sensitive sections of the Plan that
should  not be readily available to all team members, such as the
details  of  the  procurement  approach,  may  be maintained in a
separate binder.

3.3.  Evolution of the Project Management Plan Through the System
      Life Cycle

      The  Project Management Plan evolves over the course of the
system life cycle, including a subset of topics at the end of the
Initiation  phase,  and evolving into a comprehensive plan by the
end  of the Concept phase.  Exhibit 3-1 illustrates'the evolution
of the Project Management Plan through the system life cycle.

      At the end of the Initiation phase, the plan should contain
information  about  several  topics, as shown in Exhibit 3-1.  At
this   time,   only   limited  information  is  known  about  the
information management problem or the potential solutions.  Thus,
the Project Management Plan, contains some basic information about
the  project,  and  a  workplan  for the Concept phase.  Specific
topics  addressed  in  this first draft of the Project Management
Plan are:

      o  Project  Charter/Objectives - Includes identification of
         the  information  management  -problem  to be solved, the
         pertinent programmatic mission, and a preliminary view

                               19

-------
                                                                         OSWER DIRECTIVE *322a.03a
        EXHIBIT 3-1: EVOLUTION OF PROJECT MANAGEMENT PLAN

                      THROUGH THE SYSTEM LIFE CYCLE
      PHASE/STAGE
 TOPIC
                                                        cu



                                                        o
                                                        te.
 Project Charter/Objectives



 Life Cycle Adjustment



 Project Team Organization



 Project Budget



 Project Reviews/Quality Assurance



 Applicable Project Approvals



 Benefit-Cost Analysis



 Methodologies and Tools



 Workplan



 Procurement Approach



 Configuration Management Plan



 Documentation Standards



 Security Approach



 Conversion Approach



 Installation Approach



 User Support Approach



 Maintenance Approach



 Operations Approach
LEGEND;




   ART mREFINE AS NEEDED
OMPLETE m EXPAND ANDlOR ADD DETAIL  WEW ITERATION COMPLETE
                                            20

-------
                                               QSWffl DIRECTIVE ^3C2
   of  the  scope  of the problem in terms of  the  functions
   and   organizations  experiencing  the  problem.     This
   section also identifies the project sponsor.

o  Life  Cycle Adjustment - Includes any adjustments to the
   life  cycle  to  be  made  in  the  Concept  phase.   For
   example,  for  a  relatively  simple problem,   the  more
   detailed   functional  and  data  requirements   normally
   performed  during the Definition stage might  be included
   in the Concept phase.

o  Project  Team  Organization - Includes  identification of
   the  Project  Manager,  and the project participants and
   project  team  structure  for  the  Concept phase.  This
   section   identifies   participating organizations   and
   individuals  for  the Concept phase, and  also identifies
   the intended use of contractor support.

o  Project  Budget  - Identifies the total  resources needed
   to conduct the Concept phase', and includes a  preliminary
   order of magnitude preliminary estimate of the aggregate
   cost  of  all  other  stages  through the Implementation
   stage (e.g., whether the aggregate cost should be viewed
   in   terms  of  thousands,  hundreds  of   thousands,  or.
   millions of dollars, and commensurate EPA workyears).

o  Project   Reviews/Quality  Assurance  -  Identifies  the
   preliminary threshold level for the project based on the
   known   information   about   the  problem.    {How  the
   'threshold analysis' should be conducted is described in
   the  practice  paper  for  'System Life Cycle Reviews and
   Approvals'.)   Also identifies the lead reviewer for the
   project  and  a  scheduled  dates  for completion of the
   Initiation phase and Concept phase reviews.

o  Applicable  Project Approvals - Based on the preliminary
   threshold  level  for  the project and known information
   about the problem, identifies the approval authority (in
   terms of specific organization(s) and individual(s)) for
   •the Initiation and Concept phases of the project.

o  Benefit-Cost  Analysis  - Provides only a rough  order of
   magnitude  estimate  of  the project costs (based on the
   budget  estimates described above) and a brief narrative
   statement of the expected benefits and the organizations
   that  will  realize  them.    A quantitative estimate of
   benefits is not essential in the Initiation phase.

o  Methodologies   and  Tools  -  Identifies  the   analytic
   methods  and  automated  tools  .that will be used in the
   Concept phase.

o  Workplan  -  Describes  the tasks to be conducted in the
   Concept stage, noting for each task who will perform the

                         21

-------
                                                    OSWER DIRECTIVE #9028.008


         task,  its  schedule, resources to be used, and products
         to  be generated.  The Workplan should include a summary
         prepared in a Gantt chart format whenever possible.  The
         Workplan   also   identifies   at   this  time  any  key
         assumptions  or  constraints  on conducting the tasks of
         the Concept stage.
                                                    *"'
                                                    >'•
      The  Concept phase adds considerable new information to the
Project Management Plan, introducing most of the remaining topics
and  adding  detail  to  the  topics  first  addressed during the
Initiation  phase.    By  the  end  of  this  phase,  the Project
Management  Plan  is comprehensive.  The Concept phase results in
the   selection   of  a  specific  alternative  for  solving  the
information  management  problem, and the Project Management Plan
describes  the  management  approach  for taking that alternative
through the remainder of the system life cycle.

      In  succeeding  phases  and  stages, the Project Management
Plan is updated and refined as necessary, based on the results of
project activities.  Some sections of1 the Project Management Plan
may  change  significantly to address difficulties encountered in
managing  the  project.   Any changes to the basic system concept
will  likely  result  in  changes  to one or more elements of the
Project Management Plan, with the Workplan and Project Budget the.
most  likely  to  change.  If at any time it becomes necessary to
significantly change any part of the Project Management Plan, the
Project  Manager  should  retain  the prior version for reference
purposes and to support the .post-implementation evaluation of the
system.                   .

3.4   Retention of Old Project Management Plans

      The  Project Management Plan-is a living document, evolving
continuously  throughout the system life cycle.  Although keeping
the  Project  Management  Plan  current  is important, it is also
important  to  preserve  prior versions of the plan to preserve a
record of the evolution of the project.  This record will be very
useful  if there is a changeover in project management — it will
enable  the new Project Manager to more easily 'come up to speed'
on  both- the  current status of the project and its history.  'In
addition,  this record will be extremely useful if  the project is
audited  by  the  Agency's Office of the Inspector  General  (OIG).
It  will  enable  the  project  team, and the project sponsor, to
provide the information requested by the OIG much more easily and
ensure that the information provided is accurate.

      Although   the  Project  Management  Plan  may  be  refined
relatively  frequently,  the  Guidance  does not require the same
extent  of  recordkeeping  as  for  other system documents, those
contained  in system baselines.  To ensure that a complete  record
of the Project Management Plan history is retained, a copy of the
current  Project  Management  Plan  should  be  filed  with  each
approved  System  Decision  Paper,  in  the  same baseline as the
corresponding  System  Decision  Paper.   For most  projects, this

                               22

-------
                                                      QSWER DIRECTIVE £SQ£3.CCa
procedure  will result in filing a copy of the Project Management
Plan at the end of each phase and stage of the system life cycle.
    4.  RELATIONSHIPS BETWEEN PROJECT MANAGEMENT PLAN TOPICS
      The  topics  of  the Project Management Plan are related to
each  other  in  that  they address different perspectives of the
same  project  management issues.  It is therefore important that
the  contents  of  the  Project  Management  Plan  be  internally
consistent.    For  example,  the cost of contractor resources to
conduct the activities enumerated in the project Workplan must be
consistent  with  available  contract  funding  identified in the
Project  Budget.    Similarly,'  the  intended  use  of contractor
support  must  be reflected in the Procurement Approach to ensure
that   the   means  for  acquiring  such  support  (e.g.,  signed
contracts)  are  in  place  in  a  timely  manner.    Exhibit 4-1
identifies  all significant relationships among the topics of the
Project  Management  Plan.    Exhibit 4-2 describes each of  these
relationships.
                                23

-------
                                                                               OSWES DIREC
                 EXHIBIT 4-1:  SUMMARY OF RELATIONSHIPS
             BETWEEN PROJECT MANAGEMENT PLAN TOPICS


Li
Project
Ass
ie
is
Cost
S
Appro
Approac
User S
Mainte
                                                                                           1
                                                                                           •a
Project Charter/Objectives
Life Cycle Adjustment
Project Team Organization
Project Budget
Project Reviews/Quality Assurance
Applicable Project Approvals
Benefit-Cost Analysis
Methodologies and Tools
Workplan
Procurement Approach
Configuration Management Plan
Documentation Standards
Security Approach
Conversion Approach
Installation Approach
User Support Approach
Maintenance Approach
Operations Approach
      Designates two topics that address one or more common subjects,
      and that should treat these subjects in a consistent manner.  For
      example, the use of contractors as shown in a Project Workplan
      should be reflected as well in the Procurement Plan.
                                              24

-------
                                EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                                  BETWEEN PROJECT MANAGEMENT TOPICS
         Topic

         Project
         Charter/
         Objectives
                 Related Topic    Relationship
Kl
in
Life Cycle
Adjustments
                 Project Team
                 Organization
                          Project
                          Reviews/
                          Quality
                          Assurance

                          Applicable
                          Project
                          Approvals
Project Team
Organization
                          Project Budget
                          Project
                          Reviews/
                          Quality
                          Assurance
                 The lead organization for the project is
                 designated in the Project Charter.
The Project Charter designates the threshhold
level for the system, and one or more of the
organizations that should participate in project
reviews.

The Project Charter designates the threshhold
level for the system, and one or more of the
organizations that should participate in project
reviews.

The project team structure and/or participants
may change during the life cycle, and should
reflect any life cycle adjustments.  For large
systems, adjustments to the life cycle that
provide a phased development and implementation
of major modules or work packages should be
reflected in the project team organization.

The project budget, which is usually presented to
follow the standard phases and stages, should be
structured to reflect any adjustments to the life
cycle.

Life cycle adjustments which serve to combine
parts of phases and stages may affect the number
and/or timing or project reviews.  The specific
reviews to be conducted should reflect any such
adjustments.

-------
                            EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                              BETWEEN PROJECT MANAGEMENT TOPICS
     Topic

     Life Cycle
     Adjustments
     (continued)
Related Topic    Relationship
to
o»
Applicable
Project
Approvals
                      Methodologies
                      and Tools
                      Workplan
                      Procurement
                      Approach
                      Configuration
                      Management Plan
                      Conversion
                      Approach
Life cycle adjustments which serve to combine
parts of phases and stages may affect the number
and/or timing or project approvals.  The specific
approvals to be conducted should reflect any such
adjustments.

The identification of specific methodologies and
tools should be consistent with any adjustments
to the life cycle.  Also, some tools, such as
program code generators, by themselves define an
adjustment to the life cycle in that activities
conducted in one stage (e.g., Design) may
generate automatically products that are
typically produced in another stage (e.g.,
Development).

The project Workplan must be consistent with any
adjustments to the phases and stages of the life
cycle.  Each task/activity should be associated
with a specific phase or stage.

The Procurement Approach identifies the resources
to be acquired for each phase and stage, and
should be consistent with any life cycle
adjustments.

Adjustments to the life cycle should be reflected
in the products associated with each baseline.
The content of some baselines may differ from the
typical set products.

Life cycle adjustments, particularly those that
provide a phased development of modules or work
packages, may require a comparable phasing of
data conversion activities for each module.
                                                                         S
                                                                         p
                                                                         I
                                                                         m

-------
                              EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                                BETWEEN PROJECT MANAGEMENT TOPICS
       Topic

       Life Cycle
       Adjustments
       (continued)
       Project Team
       Organization
KI
Related Topic

Installation
Approach
Project
Reviews/...
Quality
Assurance
Applicable
Project
Approvals
                        Workplan
                        Procurement
                        Approach
Relationship

Life cycle adjustments, particularly those that
provide a phased development of modules or work
packages, may require a comparable phasing of
installation activities for each module.

The Project Team Organization should identify the
organizations, and individuals from each
organization, that will perform formal project
reviews and other quality assurance activities.
The designated organizations should be consistent
with the results of the 'threshold analysis'
determining the appropriate level of project
review and approval.

The Project Team Organization should identify the
organizations, and individuals from each
organization, that will provide required project
approvals.  The designated organizations should
be consistent with the results of the 'threshold
analysis' determining the appropriate level of
project review and approval.

The assignment of specific tasks presented in the
project Workplan should reflect the specific
organizations and individuals identified in the .
Project Team Organization.

The use of contractors to conduct the project is
identified in the Project Team Organization.
Each contract vehicle under which contractor
support is to be obtained, including new
procurements, should be identified in the
Procurement Approach
                                                                                                 g

-------
                            EXHIBIT  4-2: DETAILS OF RELATIONSHIPS
                              BETWEEN PROJECT MANAGEMENT TOPICS
     Topic

     Project Team
     Organization
Related Topic    Relationship
Configuration
Management Plan
                      Conversion
                      Approach
N>
oo
Installation
Approach
                      User  Support
                      Approach
                      Maintenance
                      Approach
The Project Team Organization should identify
specific organizations and 'individuals who will
perform specific configuration management
functions identified in the Configuration
Management Plan: Configuration Manager, and
Change Control Panel.

The Project Team Organization should reflect the
content of the Conversion Approach regarding the
specific organizations responsible for
determining the data to be converted to the new
system/data base, and for accomplishing the data
conversion.

The Project Team Organization should reflect the
content of the Installation Approach regarding
the specific organizations responsible for
supporting system installation, including
installations at all locations for distributed
systems.

The Project Team Organization should reflect the
content of the User Support Approach regarding
the specific organizations responsible for
providing initial system training and performing
ongoing user support activities.

The Project Team Organization should reflect the
content of the Maintenance Approach regarding the
specific organizations responsible for performing
maintenance activities, including maintenance of
interfaces with other systems and data bases.

-------
                            EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                              BETWEEN PROJECT MANAGEMENT TOPICS
                      Related Topic    Relationship
     Project Team
     Organization
     Project Budget
K>
     Project
     Reviews/
     Quality
     Assurance
Operations
Approach
Benefit-Cost
Analysis
                      Workplan
                      Procurement
                      Approach
Applicable
Project
Approvals
                      Benefit-Cost
                      Analysis
                     Workplan
The Project Team Organization should reflect the
content of the Operations Approach regarding the
speqific organizations responsible for operations
support, including support of operations at each
location for distributed systems.

The costs documented in the Project Budget should
be consistent with the costs identified in the
Benefit-Cost Analysis.

The resources identified by task in the Workplan
should be, in the aggregate, consistent with the
resources provided in the Project Budget for each
type of resource.

All contract resources identified in the Project
Budget should also be reflected in the
Procurement Approach '

The level of organization(s) responsible for
conducting project reviews should be consistent
with the results of the threshold analysis for
determining the appropriate level of project
approvals.

Benefit-Cost Analyses that show project costs
(one time investment, annual costs, and/or total
life cycle) greater than OMB and/or OIRM
reporting requirements require formal project
reviews conducted by the SIRMO and appropriate
Information Management Coordinators.

Individual project reviews and other quality
assurance activities should be specifically
identified and scheduled in the project Workplan.
                                                                                                1
                                                                                                SO
                                                                                                a

-------
                           EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                             BETWEEN PROJECT MANAGEMENT TOPICS
    Topic

    Project
    Reviews/
    Quality
    Assurance
    (continued)

    Applicable
    Project
    Approvals
Related Topic    Relationship
u>
o
    Methodologies
    and Tools
Configuration
Management Plan
Benefit-Cost
Analysis
                     Workplan
Workplan
                     Procurement
                     Approach
                     Configuration
                     Management Plan
                     Documentation
                     Standards
                      Security
                      Approach
Specific organizations and individuals who will
participate in reviews of requested changes to
the system are designated as the Change Control
Panel and are identified under both topics.
Benefit-Cost Analyses that show project costs
(one time investment, annual costs, and/or total
life cycle) greater than OMB and/or OIRM
reporting requirements require project approvals
by the Information Management Steering Committee.

Individual project approvals should be
specifically identified and scheduled in the
project Workplan.

Tasks for selecting project methodologies and
tools should be specifically identified and
scheduled in the project Workplan.

The acquisition of automated tools not currently
owned or leased by the Agency should be included
in.the Procurement Approach

The identification of automated tools, if any, to
support configuration management activities
should be consistent under both of these topics.

The selection of methodologies and tools should
be consistent with, and support, the selected
documentation standards.

The identification of automated tools, if any, to
design, implement or assess system and data base
security should be consistent under both of these
topics.

-------
                       EXHIBIT 4-2i DETAILS OF RELATIONSHIPS
                         BETWEEN PROJECT MANAGEMENT TOPICS
Topic

Methodologies
and Tools
(continued)
Related Topic    Relationship
Workplan
Conversion
Approach


Installation
Approach
                 Maintenance
                 Approach
Procurement
Approach
                 Configuration
                 Management Plan
                 Security
                 Approach
                 Conversion
                 Approach
The identification of automated tools, if any, to
support data conversion should be consistent
under both of these topics.

The identification of automated tools, if any, to
support system installation should be consistent
under both of these topics.

The identification of automated tools, if any, to
support software maintenance should be consistent
under both of these topics.

Activities for accomplishing the acquisitions
identified in the Procurement Approach should be
specifically included and scheduled in the
project Workplan.

Activities supporting 'the development of the
Configuration Management Plan, and its
implementation, should be specifically included
and scheduled in the project Workplan.
Scheduling of meetings of the Change Control
Panel should be included in the Workplan.

Activities supporting the development of the
Security Approach and its implementation, should
be specifically included and scheduled in the
project Workplan.

Activities supporting the development of the
Conversion Approach should be specifically
included and scheduled in the project Workplan.
Activities for accomplishing the conversion of
data into the new system or data base also should
be included and scheduled in the Workplan.

-------
                           EXHIBIT 4-2: DETAILS OF RELATIONSHIPS
                             BETWEEN PROJECT MANAGEMENT TOPICS
    Topic

    Workplan
    (continued)
Related Topic    Relationship
Ul
K)
    Procurement
    Approach
Installation
Approach
                     User Support
                     Approach
                     Maintenance
                     Approach
                     Operation
                     Approach
Security
Approach
                     Conversion
                     Approach
                     Installation
                     Approach
             •
Activities supporting the development of the
Installation Approach, and its implementation,
should be specifically included and scheduled in
the project Workplan.

Activities supporting the development of the User
Support Approach, and its implementation, should
be specifically included and scheduled in the
project Workplan.

Activities supporting the development of the
Maintenance Approach, and its implementation,
should be specifically included and scheduled in
the project Workplan.

Activities supporting the development of the
Operation Approach, and its implementation,
should be specifically included and scheduled in
the'project Workplan.

The acquisition of hardware and/or software not
currently owned or leased by the Agency to
provide system or data base security should be
included in the Procurement Approach.

The acquisition of hardware and/or software not
currently owned or leased by the Agency to
support data conversion, or the use 'of contractor
support to accomplish data conversion, should be
included in the Procurement Approach.

The acquisition of hardware and/or software not
currently owned or leased by the Agency to
support system installation, or the use of
contractor support to install the system, should
be ineeded in the Procurement Approach.
                                                                                                 m
                                                                                                 s
                                                                                                 ll'
                                                                                                 «•
                                                                                                 I

-------
                        EXHIBIT 4-2:  DETAILS OP RELATIONSHIPS
                          BETWEEN PROJECT MANAGEMENT TOPICS
 Topic

 Procurement
 Approach
 (continued)
Related Topic

User  Support
Approach
Configuration
Management Plan
                 . Maintenance
                 Approach
Operation
Approach

Maintenance
Approach
Conversion
Approach
                 Operations
                 Approach
Installation
Approach
Relationship
  *
The acquisition of hardware and/or software not
currently owned or leased by the Agency to
conduct training and other user support
activities, or the use of contractors to provide
user support, should be included in the
Procurement Approach.

The acquisition of hardware and/or software not
currently owned or leased by the Agency to
perform system maintenance, or the use of
contractors to maintain the system, should be
included in the Procurement Approach.

The use of contractors to operate the system
should be included in the Procurement Approach.

Procedures for assessing requested changes to the
system, including maintenance changes, and for
tracking the status of change requests, are
included in the Configuration Management Plan.
The organizations responsible for maintaining the
system should be represented on the Change
Control Panel designated in the Configuration
Management Plan.

Procedures used to record anomalies of system
operations, and to track their resolution, are
identified in the Configuration Management Plan.

Installation of the system should generally
provide for the loading of existing data prior to
cutover to full production.  The Installation
Approach should refer to the Conversion Approach
for the details of accomplishing data conversion
activities.

-------
                APPENDIX A



DETAILED OUTLINE OF PROJECT MANAGEMENT PLAN
                     34

-------
                                  PROJECT MANAGEMENT  PLAN
                                            SUMMARY DESCRIPTION
              The Project Management Plan is created during the Initiation phase and updated in
              each phase or stage of the system life cycle.  Some topics (e.g., security
              approach, maintenance approach) are summarized in the Project Management Plan,
              and presented in greater detail in other  life cycle products.
                                                  TOPICS
to
in
o   Project charter/objectives

    —  Project  identification (incorporate
        Initiation Decision Paper by reference)
    —  Mission  and objectives
    —  Scope of information management
        problem/project

o   Life cycle adjustment

    —  Consolidation of phases and stages, if
        any
    —  Partitioning of project/system into
        major work packages, modules,  etc. with
        different timing through the life cycle

o   Project team organization

    —  Project  management structure
        Manager assigned: individual,
        current organization,  authority
        Boards, committees, or other
        project management participants
        Changes or additions for Operation
        phase

—  Project team organization

        Structure and roles
        Participating organizations
        Staffing plan (including internal
        staff and use of contractors)
        Changes or additions for Operation
        phase

—  Other organizations to be  notified of
    major project events (non-participants
    in project team)
                                                                                                                   I
                                                                                                                   *
                                                                                                                   co

-------
                  PROJECT  MANAGEMENT  PLAN (Continued)
to
o   Project budget  (broken out by stage)

    ~ EPA staff
    — Contractor  services
    — Equipment acquisition
    — Hardware maintenance
    — Site preparation
    — Packaged software acquisition
    — Supplies
    — Timeshare          •
    ~ Other
    — Cost-accounting methodology

o   Project reviews/quality assurance

    — Applicable  project review level
    — Reviews to  be conducted (by stage)
    — Organization/individuals for each
       review
    — Review schedule

o   Applicable project approvals

    — Project approval level
    — Specific approvals to be obtained (by
       stage)
    — Approval organization and individuals
    — Approval schedule

o   Benefit-cost analysis (summary,  transferred
    from other life cycle products)
    —  Methodology and assumptions

    --  Benefits

           Programmatic
           Annual  monetary
           System  life

    ~  Costs

           Nonrecurring
           Recurring
           Annual
           System  life

    —  Payback period
    —  Sensitivity analysis

o   Methodologies and tools

    ~  Methodologies (non-automated)

           For Concept phase
           For Definition stage
           For Design stage
           For Development stage
           For Implementation stage
           For Production stage
           For Evaluation stage
           For Archive stage
                                                                                                          o

-------
PROJECT  MANAGEMENT  PLAN  (Continued)
          —  Automated tools/software packages

                  For Concept phase
                  For Definition  stage
                  For Development stage
                  For Implementation stage
                  For Production  stage
                  For Evaluation  stage
                  For Archive stage
                  Support required (if any) for use
                  of tools

      o   Workplan

U)         —  Concept Phase
          —  Definition Stage
          —  Development stage
          —  Implementation stage
          —  Production stage
          —  Evaluation stage
          —  Archive stage
                  Activities and  related tasks
                  Products
                  Schedule by task and product
                  Staff and contractor assignments
                  Level of resources for each task
                  and/or product
                  Task relationships/dependencies
                  Schedule of reviews' and approval
                  Performance/progress reporting
             . -    Notification

      o   Procurement approach *

          —  Resources to be acquired through
                                          existing contracts

                                              OSNER contracts
                                              Other agency contracts

                                       —  Resources to be acquired  through new
                                          procurements

                                          -   OSWER vehicles
                                              Other Agency vehicles
                                              Schedule for each procurement
                                              Norkplan for each OSWER procurement
                                              Procurement assistance individuals
                                              for each procurement

                                   o   Configuration Management Plan

                                       —  Configuration manager (organization and
                                          individual)

                                       —  Change Control Board

                                              Participants (organizations and
                                              individuals)
                                              Modification request/approval
                                              process

                                       —  Procedures/methods for configuration
                                          identification and accounting, software
                                          control,  audits

                                       —  Configuration management documentation:
                                          identification and location of existing
                                          CM logs,  and official existing baseline
                                          contents

-------
                   PROJECT  MANAGEMENT PLAN (Continued)
00
o   Documentation standards:  Standards  to be
    used for each life cycle product

o   Security approach

    —  Summary of security requirements
       (reference other life cycle products)
    —  Security organization (it applicable)
    —  Hardware and facilities measures
    —  Software and communications measures
    —  Data base security
    —  Procedural measures

o   Conversion approach

    —  Overview          .

           Data identification
           Current data location
           Organizations to accomplish   '
           conversion

    —  Manual data to be converted

           Sources
           Procedures   N
           Error conditions to be corrected

    —  Automated data to be converted

           Sources
           Procedures
           Error conditions to be corrected

o   Installation approach:  Schedule for
    installing each separately-installed system
    module

    — Dates and times, by module and location
    — Special conditions
    — Personnel to accomplish installation,
       and/or on call

o   User support approach

    — Training activities

           Materials to be prepared
           Sessions, schedules, and
           participants

    — Ongoing user support (hotline,  etc.)

o   Maintenance approach

    — Maintenance support organization
    — Release management procedures
    — Planned maintenance releases

o   Operation approach

    :— Organization of operation support
       activities
    — Reference to Operation Manual

-------
                       OSWER DIRECTIVE #9328.003
  OFFICE OF SOLID WASTE
AND EMERGENCY RESPONSE
         (OSWER)
    SYSTEM LIFE CYCLE
  MANAGEMENT GUIDANCE
    Part 3: Practice Paper
 Configuration Management
       January, 1989


-------
                                                      OSWER DIRECTIVE *302G.Kto
                        TABLE OF CONTENTS
1.  Practice Paper Purpose	    1

2.  Overview of Configuration Management Activities and
    Practices	    2
                          «

    2.1.  Configuration Management Overview	    2

    2.2.  Configuration Item Identification	    3

    2.3.  System Baseline Management  	    5

    2.4.  Change Control  	    6

    2.5.  Software Control	    9

    2.6.  Configuration Accounting  	    9

    2.7.  Configuration Audits	   10

3.  'Configuration Management Organization  	   10

    3.1.  Configuration Manager	   11

    3.2.  Change Control Board  	.'	   11

4.  Configuration Management Plan  '	   13

    4.1.  Purpose of Configuration Management Plan  	   13

    4.2.  Development and Update of the Configuration
          Management Plan	  13

    4.3.  Description of Configuration Management
          Plan Topics	  13
                                                   s


                            EXHIBITS
2-1.  Configuration Management Through the System
      Life Cycle   	    4

2-2.  Contents of  System Baselines   	    7

4-1.  Evolution of Configuration Management Plan
      Through the  System Life Cycle	   14

4-2.  Outline of Configuration Management Plan	   15

-------
                                                     OSWER DIRECTIVE 3SOSE.C03



                    1. PRACTICE PAPER PURPOSE


      This  practice paper constitutes a section of Part 3  of the
Office  of Solid Waste and Emergency Response (OSWER) System Life
Cycle  Management  Guidance.    It  describes OSWER's practice of
configuration management, tailoring industry proven configuration
management methods and techniques to the OSWER environment.

      Configuration' Management  is  a  function  which serves to
systematically identify the characteristics of a system (referred
to as "configuration items"), and formally control any changes or
additions  to  those  items.    Examples  of  configuration items
include  specific  functional  and  data requirements, the system
design, and documentation such as the User Manual.  Configuration
Management  helps maintain the integrity of the system throughout
its  life  cycle,  and facilitates communication about the system
among  system  project  team members, users, and other supporting
organizations.

      This   practice   paper  provides  the  following  guidance
regarding the implementation of configuration management:

      o  Describes    specific    activities    associated   with.
         configuration management;

      o  Describes  project organization structures  to accomplish
         configuration management; and

      o  Describes    the   .documentation   of   project-specific
         configuration  management  activities in a  Configuration
         Management Plan.

      Ensuring  the  integrity  of the system may be particularly
challenging  for  a  large and/or complex system.  A rigorous and
disciplined   configuration  management  function  will  maintain
system  integrity  for  these  systems,  and all other systems as
well, in the following ways:

      o  .Helps  ensure  clear identification and documentation of
         functional and data requirements;

      o  Helps   ensure   that   all  approved  requirements  are
         traceable through the system design;

     •o  Provides a means to unambiguously  identify  and reference
         the  specific  components  of  the system design  and the
         actual    system    (e.g.,   subsystems,    data    bases,
         documentation);

      o  Helps  ensure  that  an  adequate  review   of  requested
         changes  to  the  system,  and approval by  an authorized
         organization  and  individual,  takes  place  before the
         system is changed;

-------
      o  Helps  ensure  effective  control  over  changes  to the
         software and release of changes to users;

      o  Provides  a  means to clearly identify the status of the
         system, and of individual system components, at any time
         throughout the life cycle;

      o  Helps  ensure  the  consistency of products, such as the
         agreement   of   documentation   with  the  applications
         software and other components of the system;

      o  Facilitates effective communication among system project
         team  members, users, and other supporting organizations
         about  the  characteristics of the system and its status
         throughout the system life cycle; and

      o  Provides  a  means  to  reconstruct the evolution of the
         system, and rationale for significant decisions, through
         the prior phases and stages of the system life cycle.

      This  practice  paper  is  not intended to provide detailed
configuration  management  procedures.   Rather, it describes the
activities,  organizational  framework,  and  types of procedures
that  should  be  adopted  by  each  system project for effective.
configuration  management.    The details of the procedures for a
given  project  should  be  developed  on a case by case basis to
reflect  the  program  needs, technical environment, and involved
organizations for each project.                       '  '


             2. OVERVIEW OF CONFIGURATION MANAGEMENT
                    ACTIVITIES AND PRACTICES


2.1.  Configuration Management Overview


      Configuration  'management  encompasses  six sets of related
activities:
                                                   t
      o  Configuration  Item  Identification which identifies the
         characteristics  of  the  system  to  be  controlled  by
         delineating specific configuration items;

      o  Baseline  Management, which establishes repositories (or
         libraries)  that  contain  the  documentation  of  these
         items;

      o  Change  Control,  which  ensures  that only, advantageous
         changes  are  made  to  the  system  as it  exists in the
         controlled baselines;

-------
                                                   OSWcR DIRECTIVE iSO'i5.Sr:3


      o  Software  Control/   which preserves  the  integrity  of the
         software while  changes  are being  made;

      o  Configuration  Accounting/  which monitors  the status  of
         theconfigurationitems,  and requested changes  to the
         system   in   terms  of   their    impact   on   specific
         configuration items; and

      o  Configuration  Audits/   which confirm the proper working
         of the configuration management function and help  ensure
         that system documentation is complete and current.


      These  activities   are performed throughout the system life
cycle/  starting with the Initiation phase and continuing through
the  end  of system operations in the Archive stage.  Exhibit 2-1
provides  an  overview of configuration management throughout the
system life cycle.

2.2.  Configuration Item Identification

      Configuration   item   identification   serves  to   clearly
delineate   the   significant   characteristics  of   the   system/
providing   a   common   language/  or  referencing   scheme/  for.
describing  the  system.    It  is  the  delineation  of  specific
configuration  items that will enable all individuals involved in
the evolution of the system to communicate effectively/  using the
common  language.    Examples  of  such  characteristics   include
specific    functional    and    data    requirements/   specific
characteristics  of  the  system design (e.g./ subsystems/  files/
(etc.)  and system documentation.  Each characteristic,  or  set of
related  characteristics/ is- referred to as a configuration item.
The  delineation  of individual  configuration items  can  be. viewed
as creating the 'labels' for the characteristics described in the
conventional  documentation  of   the  system.    For example, the
design  for  a system's application programs that produce reports
for  use  by regional offices may be delineated collectively as a
single configuration item called 'Regional Report Programs'.

      When . conducting configuration item identification/ be sure
to note the following practices:

      o  Configuration  items delineate different types  of system
         characteristics, and reflect the current phase  and stage
         of the life cycle.

         —  Individual functional and data requirements/ or sets
             of  related  requirements,  are  delineated  in  the
             Definition stage;

         —  Major   attributes    of   the   system  design,  are
             delineated in the Design stage,  such as:

-------
              EXHIBIT 2-1: CONFIGURATION MANAGEMENT
                     THROUGH THE SYSTEM LIFE CYCLE
                                                                     Initiation Baseline
                                 LOO/RESPOND TO CHANGES
DeTmitioo Btulioe
Deiip Bateline
Development Baseline
                                                                                              Definition Baseline

                                                                                              |iprf.|ftd-
                                                                                              Initiation Baseline
                                                                                       Iniliatioo Bauline
                                                                                       Definition Bucline
Devdofnieni Boeline
Opcfational Bucline
r-   i
                             Inilitlioo Baieline
                             Definition Baseline
                             Design Baseline
                             Development Baseli
Updated:
Initiation Baseline
Definition Baseline
Design Baseline

-------
                                                    OSWER DIRECTIVE ^u^.
             -  Hardware  and  technical  environment(s)
             -  Logical data  base  structures
             -  Physical  data base structures
             -  Applications  software
             -  Utility software

         —  The  application  software (i.e., programs),  such as
             input  and   update processing,  reporting,  and system
             administration   processes  are  delineated   in  the
             Design stage.

         —  Specific  system documentation,  such as  User  Manual,
             Maintenance    Manual,   and  Operations   Manual  are
             delineated  in the Development stage.

      o  Configuration  items  are delineated and recorded in the
         normal documentation of  the system life cycle. An index
         of  configuration  items   should  be  included  in  each
         documentation product.

         —  The index identifies  specific configuration  items.

         —  The  index   also  identifies  specific relationships
             between  configuration  items  in   the  same  product.
             (e.g.,   between   different  parts of  the   system
             design),  and  also  relationships with  previously
             defined  items  (e.g.,  between design configuration
             items and requirements configuration  items).

         —  The  index  helps  confirm  the  traceability of the
             system  throughout  the  life  cycle,  and is  a vital
             tool to accomplish system audits.

2.3.  System Baseline Management

      System  baselines  are  collections of life  cycle products,
including  hardware and software,  as well as the documentation of
the  system.   An easy way to understand what a baseline is is to
view  it  as  a  controlled  library  collection.     Creating and
managing,  .baselines   does   not   require  the  development  of
significant  additional  system  documentation.  System baselines
are  established  throughout  the life cycle to establish a clear
basis  for  monitoring the status and progress of  the system, and
to   facilitate  communication  (particularly  for   large  system
projects).

      Five  baselines  are  developed, and maintained, throughout
the life cycle:

      o  Initiation  Baseline  -  Contains  documentation  of the
         information management problem,

-------
                                                   OSWER DIRECTIVE *S023.03$


      o  Definition  Baseline  -  Contains  documentation  of the
         functional and data requirements,

      o  Design   Baseline   -   Contains  documentation  of  all
         attributes of the system design/

      o  Development   Baseline  -  Contains  the  automated  and
         hardcopy     products    of    development    (including
         documentation),   and   of   system  changes,   prior  to
         installation in the user environment, and
      *•

      o  Operational   Baseline  -  contains  the  automated  and
         hardcopy products for the currently installed  system.

      Each  baseline  contains different life cycle products, and
provides  a  different  perspective  of  the system. Exhibit 2-2
illustrates   the   content   of   each   baseline.     Important
characteristics  of  baselines  and their development include the
following:

      o  Baselines contain approved products only.  Project teams
         should  use other libraries to monitor the status of and
         control  products that are under development and not yet
         approved.

      o  Baselines  contain  the  product  as first approved, and
         subsequent approved changes.  For documentation, changes
         may  be added to the baseline in the form of an entirely
         new  document,  a replacement document, or as  an addenda
         or replacement pages.

      o  All  baselines are maintained throughout the life of the
         system,  not  just  the  operational  system.     If  the
         information  management problem changes, or the approved
         requirements  for the system or system design  change, it
         is  essential  to  keep all of the established baselines
         current  in  order to have available complete, accurate,
         documented information about all aspects of the system.

2.4.  Change Control

      Change  control  is  a  formal process for determining what
changes  are  to  be made to the system.  Change control requires
the  documentation  of specific requests for modifications, and  a
review of the requested modifications, and consideration of their
impact   on   the  system,  before  they  are  made.    The  term
'modifications' refers both to changes and to minor enhancements.
Change  control  determines which requested modifications will be
made and which will not be made.      •  .

      Change control applies to modifications requested after the
system  concept is approved and before the system  is implemented,
as well as to requests to modify fully operational  system.  For  a
system under development, the process for approving changes

-------
EXHIBIT  2-2:  CONTENTST)F SYSTEM  BASELINES
   /System Concept       \
Initiation Dccliion
              Paper  \
         ZAcctfUoc* Tea Dncumeni
       	,	_	.

    / Sygem Tea
                                                                          A)K» Support Maieiiali (Dev.V


                                                                       /Security Manua
                                                                      /Oftiuiont Mtnu.l
                                                                      KUinlcnmce Manual (Dev.)V
                                                                     	       v—•»
                                                                    liter Manual (DevelopmeniA
                                                                             /Uicr Support         V

                                                                             Matcriali (Prod)       \

                                                                        /Security Manual (OpcXProd )\
                                                                      /OpctaUooi Manual (l*rod)~X
                                                                  /MainlCTance Manual (Prod.) \"


                                                                  Uier Manual (Production)  V
                                                                                                          ill
                                                                                                          S';
                                                                                                          III
                                                                                                          I)
                                                                                                          IVI

-------
                                                     OSWER DIRECTIVE *SC23.COa


may  be  integrated  with  the formal approval  process  that takes
place at the end of each stage of  the life  cycle.   However, since
these  formal  approvals are occur at the end of  the stage, it is
better  to  establish  a  separate  change   control  process,  and
include  a summary of changes approved using this  process as part
of the System Decision Paper submitted for  formal  approval at the
end of each stage.

      Other  important  aspects of  change control include the
following:


      o  Change  control  provides  an  examination  of requested
         changes  to  the  system   before   the  system is changed.
         This  examination  is  referred to as a  'Change Request
         Impact Analysis'.  This analysis considers all pertinent
         characteristics of the change:

         —  Functional and/or data requirement,
                        ;
         —  Reason  for  the  change  (e.g.,   new  regulation or
             program policy),

         —  Benefits of making the change,

         —  Impact  of  not  making  the change  on OSWER program
             operations, and .the associated organizations,

             Specific  system configuration items  impacted by the
             change, and extent of impact,

             Impact  of  the  change  on related  systems and data
             bases,

         —  Cost  of  accomplishing  the change,  (including both
             onetime  costs,  and  recurring costs  [operations and
             maintenance], and expected source  of  funds),

         —  Impact of the change  on user organization  procedures
             and/or staffing requirements,

         —  Timeframe for accomplishing the change,

         —  Potential  risks  in   making   the   change   (i.e., is
             successful change a certainty?),

         —  Disposition of prior  requests  for  the same change.or
             a comparable change.

      o  Requested  changes  for  a  system under development may
         apply  to  system characteristics  that have not yet been
         approved  and  recorded  in  a baseline.    The Project
         Manager  should  design a process  for  determining how to
         handle  these  requests,  and document  the process in the

                                8

-------
                                                       OSWER DIRECTIVE #3023.00:
         Configuration  Management   Plan.  The  process may  be the
         same  one used to decide how to handle requested changes
         to previously approved configuration items.


2.5.  Software Control

      Software  control is a set of procedures  to ensure that the
integrity  of  the  system is preserved when approved changes are
being  made  to  the  system,  or  in the  event of a  disaster and
restoration of the system is needed.  Software  control procedures
are  particularly  important  during  the  Production  stage  of the
life  cycle.    Software  control  ensures  that  changes  to the
computer  programs  are  developed  and tested using a copy  of the
programs and a test data base, and  do not  adversely affect  system
users.    However,  software control is also important during the
Development  and  Implementation stages to control changes  during
the  initial  building  and installation of  the system.   For each
system, the software control procedure should cover the  following
points:

      o  Describes    how   the  . development    and   operational
         environments of the system will be segregated.

      o  Describes  the  procedures  to be  used  to install new
         versions  of the software  in the  production  environment,
         including  procedures  to ensure  that  a system installed
         at    multiple   locations   '(e.g.,   regional    logical
         mainframes,   multiple   personal  computers,  etc.)  is
         properly  integrated  and  distributed  data  bases  are
         effectively synchronized.

         Other  topics  may  be  included   as  appropriate to the
system.

2.6.  Configuration Accounting

      Configuration Accounting is an administrative procedure for
maintaining  system  baselines  and  monitoring the status of the
system  throughout  the  life  cycle.  Important elements of this
procedure include:

      o  Logging  and  storing  each  life  cycle  product in the
         appropriate baselines upon the approval of the product.

      o  Recording  all  requested  changes, maintaining a log of
         change  requests,  and also recording the disposition of
         each  request  (e.g.,  approved,   held  pending  further
         analysis, disapproved).

      o  Monitoring the status of approved changes, and recording
         completed changes in the appropriate baseline.   Specific
         tracking  points  are  determined for  each system and/or
         change as appropriate.

-------

      o  Providing  the  records used to accomplish configuration
         audits.

         The  Configuration Accounting procedure may also include
activities  for producing copies of approved and baselined system
documentation   for   use  by  project  team  members  (including
contractors) throughout the life cycle»
      it
2.7.  ''Configuration Audits

      Configuration  Audits  are examinations of the products and
related documents submitted for inclusion in a baseline to assure
that   they  are  complete,  clearly  presented,  and  internally
consistent.    This  examination  is  oriented  to  adherence  to
guidance  and standards.  These audits support formal reviews and
evaluations  of the system by ensuring that required products and
documents  are  complete  (e.g.,  meet identified standards), and
provide   effective   traceability  to  related  products.    (Of
particular   note,  audits  do  not  evaluate  qualitatively  the
programmatic  and/or  technical  content of the product.  That is
done  by  formal reviews and other quality assurance activities.)
Audits help ensure that the resources used to conduct reviews and
evaluations  are  not  applied to products that are not yet ready
for the review.

      Audits   also  help  confirm  the  proper  working  of  the
configuration  management  function    -.-  they  help ensure that
configuration  accounting  records  are complete and current.  As
illustrated  in  Exhibit 2-1, audits are conducted throughout the
life cycle.

      In general,, one audit is conducted in each of the following
stages:

      o  Concept
      o  Definition
      o  Design
      b  Development
      o  Implementation

      For    the   Production   stage,   audits  ' are   conducted
-periodically,  usually  annually  or every other year, during  the
Production stage.  However, more frequent audits may be conducted
in  any stage to ensure that deficiencies found by a recent audit
have been corrected.


            3. CONFIGURATION MANAGEMENT ORGANIZATION
                                  *             •
      Implementation   of   effective   configuration  management
requires  the  formal  designation  of  configuration  management
responsibilities for each system project.


                                10

-------
3.1.  Configuration Manager

      Each  system  project  will  have  a  single  Configuration
Manager.  The Configuration Manager establishes  and maintains the
configuration management records for the system.  This  individual
should  be designated during the Concept phase,  and should report
directly to the Project Manager.  For a small system,  the Project
Manager  may serve as the Configuration Manager.  Specific duties
of the Configuration Manager should include:

     *'o  Maintaining  a  complete  file  and  log of   all change
         requests,    including   requests   to   modify   system
         characteristics that are not yet approved and baselined,
         as  well  as  requests  to modify approved  configuration
         items.

      o  Recording   the  disposition  of  all  change  requests,
         including   approval/disapproval,   and  completion  and
         implementation of the change.

      o  Preparing  periodic  reports of configuration status, as
         needed,

      o  Providing  physical control over baselined documentation,
         (e.g.,   retaining  at   least  one  official  copy of the
         documentation), and

      o  Providing  assistance   and support to the performance of
         configuration audits.

      The   Configuration   Manager    is   accountable    for  the
completeness   and  integrity  of  the  configuration  management
records,   and    should    be  an  EPA  employee.    However,  the
Configuration  Manager  may  obtain  staff  support  using   other
members  of  the  project  team, including contractor support, as
appropriate.

      The  Configuration   Manager  usually does  not have  specific
decision making authority.
         "                                         /
3.2.  Change Control Board

      Each  system  project will  establish a Change Control Board.
Change  Control   Boards  examine requested  changes to  the system,
direct  the change  request impact  analysis  (an analysis  conducted
by  the  project  team)  and, based on the  results, determine the
changes  that are to be made and those that are  not to be made  to
the  system.    Several  important  issues  should  be considered
carefully when establishing a Change Control  Board:

      o  The  Change   Control Board  should  be established as soon
         after    the   approval  of   the  System Concept as  is
        . practical,  and   no  later   than the start of the "Design
         stage.     The. Board   reviews  requested  changes to the

                                11

-------
    concept,   requirements,   and design to the system during
    the  evolution   of   the   system,  as  well  as requested
    changes to an operational system.

 o  The Change Control Board  may act in an advisory capacity
    to  the   Project Manager,  or  may  serve as a decision
    making  body.     The specific authority of the Board is
    determined  for   each  system,  and is documented in the
    Configuration Management  Plan.

-o  The  Change Control Board  is  usually  chaired by the
    Project Manager, but may  be chaired by any individual.

 o  The  composition of the Change  Control  Board should
    reflect   those   organizations  directly  affected by the
    system  (e.g.,   providers of  data, direct users of the
    software,  and users  of the system outputs).

    —  Systems which   affect multiple OSWER offices should
        include a member of each office on the Board.
                                                         «•«.

    —  Systems which   affect the  Regional offices and/or
        state   agencies  should  include  at  least regional
        representatives  on   the  Board,  and should include.
        State  representatives if appropriate.

    —  All   Level   I systems should include representatives
        of  OIRM and of  the OSWER SIRMO.  (Refer to practice
        paper   for Reviews and Approvals for definition of  a
        Level  I system).
  *
    Specific   individuals  included  on  the Board should be
    determined  jointly  by   the  Project  Manager  and  the
    appropriate program  manager(s).

 o  The  Change Control Board  may operate under different
    names,  such as  'System Advisory  Group',   'Board  of
    Directors',  and   'Configuration   Management  Board'.
    However,  the name Change  Control Board is preferred.
               »                                   .
 o  The size  and composition  of the Change Control Board may
    change over the  life of the system.

 o  Not all requested changes need be examined by  the Change
    Control Board.   The  Board may designate certain types of
    simple, low impact  changes that can be approved directly
  ;  by the Project Manager.

 o  A project may elect  a higher  level of review  for certain
    types of  requested changes.' The Change Control  Board
    may  request that these changes be reviewed by the  OSWER
    Information Management Steering Committee.
                          12

-------
                4. CONFIGURATION MANAGEMENT PLAN


4.1.  Purpose of Configuration Management Plan

      The    Configuration    Management   Plan   describes   the
organization  approach  and  specific  procedures  to  be used to
implement  configuration  management  for  the  system.   It also
identifies  the  physical  location  of  system  baselines — the
locations  in  which  hardcopy  documentation  is stored, and the
automated  libraries  used  to  store other documentation and the
software components of the system.

      It  should  be noted that the Configuration Management Plan
provides  information  that unauthorized individuals might try to
use  to  tamper  with  the  applications  software.  For security
reasons,  the identification of automated libraries may be stored
external to the Configuration Management Plan, in a document that
is less accessible, in order to avoid disclosing such information
to unauthorized individuals.

4.2.  Development and Update of the Configuration Management Plan

      The  Configuration  Management  Plan  is a component of the
Project  Management  Plan.    It  may  be included in the Project
Management Plan document, or may be developed and maintained as a
separate document.  The configuration accounting documents should
generally  be  maintained  separate  from  the Project Management
Plan,  but  may  be  combined  for very small, simple systems for
which few changes are anticipated.

      The  Project  Manager  and  Configuration Manager have lead
responsibility  for  development  of the Configuration Management
Plan.

      The Configuration Management Plan is initiated early in the
life  cycle,  and  all  sections  are completed by the end of the
Design  stage.    However,  the  Configuration Management Plan is
continually  refined and updated throughout the system life cycle
as needed.

      Exhibit  4-1  provides  an  outline  of  the  Configuration
Management Plan and illustrates its evolution throughout the life
cycle.

4.3.  Description of Configuration Management Plan Topics

      The Configuration Management Plan should follow the general
outline and cover the topics presented in Exhibit 4-2.
                               13

-------
                                EXHIBIT 4-1:
       EVOLUTION OF  CONFIGURATION MANAGEMENT PLAN
                   THROUGH THE SYSTEM LIFE CYCLE
         PHASE/STAGE
   TOPIC
Introduction

Configuration Management Organization

Configuration Item Identification

Change Control Process

Configuration Accounting

Configuration Audits
                      v
System Baselines

Software Control
 I.EfiEND;
                                         EXPAND AND/OR ADD DETAIL
REFINE AS NEEDED
                                                                                    il

-------
                                                 G3WEF; D!F.2mVE FS22
                     EXHIBIT 4-2
      OUTLINE OF CONFIGURATION MANAGEMENT PLAN
o  Introduction

   —  Identification of the system
                                                *         v.
                                                   j
   —  Purpose  and  scope  of the Configuration Management
       Plan                       ~;.               j

   —  Related systems affected by Configuration Management
       Plan

o  Configuration Management Organization

   —  Project Manager
       :  •     i          •      f
   —  Configuration ..Manager ' j    -

       -": Individual a"hd home organization     •••••  -
         i '    '          •
   —  Change Control Board

     -.*'•-.' Role and Authority               	
       -  Chairperson and home organization
      /-  Other participants and  home  organizations
      ' -  Staffing and  contractor support (if applicable)

o  Configuration Item Identification        	      ^  ?: •*
      •*    >                 ,'*                           "
   —  Procedure   or   method    for   configuration '"item
       identification      ;                    ,   ?    ..,  -;; ^
                       '•;-":   •           .   .,,.  „.      •' .-.  t 'v>

   —  Configuration  item   indexes  for  each  baseline;1—
       lists  of  configuration   items and cross-references
       among  items   (The   indexes  may be an attachment to
       Configuration Management Plan)

o  .Change Control Process    i      .            ^       ^  *«

   —  Process overview

   —  Change request procedure and form (if applicable)

   —  Change request examination criteria

   —  Levels of examination and  approval

   —  Documentation of examination results and approvals
                          15

-------
                EXHIBIT 4-2  (continued)
       OUTLINE OF CONFIGURATION MANAGEMENT PLAN
 o  Configuration Accounting

  *"--  Overview of  accounting  process     ?-
T'  i '  '       ''••..•                    '  '}
 ^  -r-  Change^request  status accounting procedure

    —  Configuration item  status  forms and procedure
-.;   . -             ^ •'•    •! - f •   .-»(•• - • -,.      -,
    —  Configuration status reporting procedure

    —  Automated tools used to support status  accounting
 *T
-------
                                          OSWER DIRECTIVE #902B.OOa
  oo  Contents of  software release package -
      software, documentation
  oo  Quality control  of  release  packages
  oo  Transmission or  transmittal of package to
      each  installation
  oo  New release  installation assistance,

•  Controls  to ensure effective system integration
  and data  base synchronization upon installation
  of new software

  oo  Synchronization    of   data   base   at  each
 -or,-?.;, installation7  " l:
  oo  Synchronization     of    data   base   across
      installations for same 'system
9_op  Synchronization   of  data  base  with related
" ~c: ":data  in other systems  '
                  17

-------