, Environmental Protection Agency $ «» \ Office of Pesticide Programs ° Detailed Design for an Information Processing System for the Pesticides Monitoring Program VL Operations Manual December, 1978 Contract No. 68-01-3833 ARTHUR YOUNG & COMPANY ------- OPP MONITORING SYSTEM DETAILED DESIGN DOCUMENTATION The Detailed Design Documentation for the EPA/OPP pesticides Monitoring System consists of 7 Chapters vrhich are bound in 6 volumes. These chapters are: Chapter Title I System Overview Bound Together II Implementation Plan III Data Base Design IV Input/Output Specifications V Functional Module Specifications VI Operations Manual VII User's Manual Each of these chapters is largely self-contained but may refer to information contained in other chapters as necessary. ------- TABLE OF CONTENTS TOPIC PAGE 1. DATA BASE GENERATION VI-1 2. APPLICATION PROGRAM SUPPORT VI-16 3. DATA BASE LOADING VI-23 4. DATA BASE UPDATING VI-26 5. BACK-UP AND RECOVERY VI-27 6. DATA BASE MAINTENANCE VI-41 7. SECURITY PROCEDURES VI-44 8. ARCHIVAL DATA VI-46 ------- VI. OPERATIONS MANUAL ------- VI. OPERATIONS MANUAL INTRODUCTION The OPP Pesticides Monitoring System Operations Manual integrates PMS specific topics such as database update and archival procedures with standard IDMS database system procedures. The information pertaining directly to IDMS routines, syntax and Job Control Language has been abstracted from Cullinane Corporation IDMS manuals and tailored to OPP's specific needs. The manuals referenced are Database Design and Definition and Installation and Operations Guide. In order to tailor the JCL to OPP's operating environment, the following names have been used throughout the manual. SYS2.IDMS.OPPMON.DIRECTRY - the DSNAME for the IDMS data directory. SYS2.IDMS.LOAD - the DSNAME for the library containing IDMS system modules as delivered by Cullinane; resides on APPL01. SYS2.IDMS.OPPMON.SUBSDMCL - the DSNAME for the library containing Subschema and DMCL load modules which belong to the OPP monitoring data base; resides on APPL02. SYS2.IDMS.OPPMON.DBPROC - the DSNAME for the library containing Data Base Procedure load modules which are used by the OPP monitoring datbase; resides on APPL02. SYS2.IDMS.OPPMON.PROG - the DSNAME for the library containing application programs which use the OPP monitoring data base; resides on APPL02. VI-1 ------- 1. DATA BASE GENERATION (1) Data Directory Generation Functional Description The IDMS Data Directory System is a communications tool designed to provide database control and documentation. All of the IDMS compiler and processor system components operate in conjunction with the IDMS Data Directory file (Figure VI-1). This file (which itself is an IDMS database) represents a definitional resource in which IDMS stores all information concerning the user's database environment. Since all major system components, including the Schema, Subschema, and DMCL compilers, the Data Directory Utility Program (IDMSCLUC), and the PL/I, and DML-COBOL processors, post information into the Directory, it becomes a centralized information resource for database administration. For this purpose, the Data Directory Reports Program (IDMSRPTS) produces reports which are defined in detail in 2. Application Program Support. Allocation The Data Directory is allocated as an OS file prior to the initialization step. The allocation should be done in a separate step in order to avoid changing the JCL for the initialization step prior to re-compiling the schema. The Data Directory is allocated using the following JCL. // EXEC PGM=IEFBR14 // DD DSNOPPMON, UNIT=3330-1,VOL=SER=USRVOL, // SPACE = (960,1000),DISP=(NEW,CATLG) OPPMON = Data set name of Data Directory USRVOL = serial number of disk containing Data Directory Initialization - IDMSINIT Since the Data Directory is itself an IDMS database,IDMSINIT must be the first program run prior to processing. This step is mandatory, since each page of the data directory must be formatted prior to storing records. Consequently, IDMSINIT performs the following initializing actions: VI-2 ------- IOMS OOL COMPILERS 1DMS OML PROCESSORS IOMS DATA DIRECTORY IDMS DIRECTORY REPORTS I QMS/CULPRIT FIGURE vi-i -,= IOMS DATA OISECTCRY FILE ------- Clears and formats direct access blocks (one per page) for the Data Directory Produces a parameter listing General information on the IDMSINIT function can be found in the IDMS Database Design and Definition manual, page 11- 4. The following card formats are specific to the Initialization of the Data Directory: PROCESS = TOTAL, DMCL = IDMSBASE The DMCL name that provides access to the entire database directory network is always IDMSBASE. This formatting operation must be performed prior to the execution of the IDMS Schema Compiler. Figure VI-2 illustrates the initialization of the IDMS Data Directory. The JCL for performing the operation is as follows: // EXEC PGM=IDMSINIT //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSLST DD SYSOUT=A //SYSDDL DD DSN=SYS 2.IDMS.OPPMON.DIRECTRY,DISP=OLD //SYSIPT DD * PROCESS =TOTAL,DMCL=IDMS BASE /* SYS2.IDMS.LOAD = data set name of IDMS load library SYSOUT=A = printed output file SYSDDL = DD name of Data Directory SYS2.IDMS.OPPMON.DIRECTRY = data set name of Data Directory (2) Schema Compilation Functional Description A schema is a 'global' definition of the database, including descriptions of all files, areas, records, and sets in the database environment. Defined by the DBA staff, the purpose of the schema is two-fold: To provide commonality of data descriptions and data relationships among all user programs accessing the database VI-3 ------- pagers/ •OOLDML*AflEA LOGICAL PAGE NUMBERS (LPN'S) *OOLDM(. IS THE NAME OP Trig IQMS DATA OlflECTOaY AflEA RELATIVE BUOCX NUMBEHS (RBN'S) FIGURE vi-2 INITIALIZING THE DATA DIRECTORY ------- To maintain data-program independence When the user has coded all necessary Schema DDL statements for his database description, he is ready for actual execution of the IDMS Schema Compiler. Figure VI-3 shows the components of the schema phase of IDMS data description. Syntax and Semantics The following paragraphs describe the general function, format/order, and rules governing SCHEMA DESCRIPTION statements. Specific information for each statement can be found in the IDMS Database Design and Definition manual, 6.3. General Function SCHEMA DESCRIPTION statements identify the schema according to: Schema Name, Author, Date, Installation, and Remarks File Description Area Description Record Description Set Description. The statement groups must appear in the sequence shown above, each describing one aspect of the whole schema, and each relying on the content of O-he description statements from the prior group(s). Each statement group begins with a descriptive paragraph name. Processing of statements for a specific group begins with the associated paragraph name and ends with the recognition of the paragraph name of the next group or the end of the statement input stream. At least one entry in each statement group, except the set group, is required. JCL IDMSCHEM // EXEC PGM=IDMSCHEM,REGION=180K //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSDDL DD DSN=OPPMON,DISP= (OLD,PASS) VI-4 ------- SCHEMA SOURCE 1DMSCHEM DATA DIRECTORY ERRORS & WARNINGS INPUT STATEMENT UST1NG FIGURE vi-3 DEFINING SCHEMA ------- //SYSJRNL DD UNIT=TAPE62,VOL=OPPDIR,DSN=OPPJRNL,DISP=MOD //SYSOUT DD SYSOUT=A //SYSLST DD SYSOUT=A,DCB=BLKSIZE=133 //SYSIPT DD * Schema DDL source statements (omit if SYSIPT is other than *) SYS2. IDMS. LOAD = data set name of IDMS load library SYSDDL = DD name of Data Directory OPPMON = data set name of Data Directory SYSJRNL = DD name of Journal file SYSOUT=A = printed output files * = input unit (3) DMCL Compilation Functional Description The Device-Media Control Language (DMCL) is used to: Control those data sets that are on-line Regulate user page (block) size for the database Specify the buffer pool space used by IDMS at execution time. The DMCL Compiler cannot be executed until an error-free schema compilation has been achieved and the associated Data Directory File (as built by the Schema Compiler) is available. An object DMCL module encompasses all or selected areas of the user's database and provides: Necessary file control block information (DTF's, DCB's) to permit IDMS run-time access to the user database files on disk and to write the system journal (log) file Mapping tables to allow IDMS to translate logical page addresses to proper relative block assignments within the secondary storage file(s) Information for establishment of system buffer pools (in OS, main storage is dynamically acquired by IDMS at execution time) Figure VI-4 traces the significant steps in executing the DMCL phase of data definition. Syntax and Semantics VI-5 ------- The DMCL consists of the following three statement groups: Device-Media Description statements Buffer Section statements Area Section statements Journal Section statements The statement groups must be coded in the order shown: DEVICE-MEDIA DESCRIPTION statements identify the DMCL Name, Author, Date, Installation, and Remarks The BUFFER SECTION statements are used to assign a name to each buffer pool, declare the page size (block size) of pages in the pool, and declare the number of pages (blocks) in the pool. The AREA SECTION statements specify the logical areas defined in the Schema DDL that are to be available to the DBMS through this DMCL module. The JOURNAL section provides information about the journal file. It is mandatory for disk journaling, and optional for tape journaling. If the JOURNAL section is present in the DMCL, the journal file need not have been described in the schema. If it is missing from the DMCL, tape journaling as described in the schema will be used. If journal-file specifications are included both in the DMCL and the schema, those in the DMCL override the schema description. &• Specific rules concerning each section statement can be found in the IDMS Database Design & Definition manual. Execution Instructions; To define a DMCL module, execute the program IDMSDMCL followed by the assembler and linkage editor. Input to IDMSDMCL consists of a source deck written in DMCL DDL. Output is as follows: A description of the DMCL module stored in the Data Directory An assembler source program and device-media input listing Input to the assembler consists of the source program produced by IDMSDMCL. Output is a relocatable program and an assembler listing. VI-6 ------- Input to the linkage editor consists of the relocatable program produced by the assembler. Output is an object program and a linkedit map. The following JCL is reflected in Figure VI-4: IDMSDMCL //STEP1 //STEPLIB //SYSDDL //SYSJRNL //SYSOUT //SYSPCH //SYSIPT DMCL SOURCE //STEP2 //SYS PRINT //SYSLIB //SYSUT1 //SYSUT2 //SYSUT3 //SYS PUNCH //SYSIN //STEPS EXECPGM=IDMSDMCL,REGION=160K DD DSN=SYS2.IDMS.LOAD, DISP=SHR DD DSN=SYS2.IDMS.OPPMON.DIRECTRY,DISP= (OLD,PASS) DD UNIT=TAPE62,VOL=OPPDIR,DSN=OPPJRNL,DISP=MOD DD SYSOUT=A DD DSN=&&DMCL,UNIT=SYSDA,DISP= (NEW, PASS), SPACE = (80,(400,40)),DCB=BLKSIZE=80 DD * STATEMENTS OMIT IF SYSIPT is other than*) EXEC PGM =IEUASM,PARM='NOLOAD,DECK',REGION=80K DD SYSOUT=A DD DSN=SYS1.MACLIB,DISP=SHR DD UNIT=SYSDA,SPACE=(CYL,(2,2)) DD UNIT=SYSDA,SPACE=(CYL,(2,2)) DD UNIT=SYSDA,SPACE=(CYL,(2,2)) DD DSN=&&ASMBL,UNIT=SYSDA,DISP= (NEW,PASS) SPACE=(80,(400,40)) DD DSN=SSDMCL,DISP= (OLD,DELETE) EXEC PGM=IEWL,PARM='SREF,LIST,LET,SIZE=(96K,12K)1, REGION=96K //SYS PRINT DD //SYSLIN DD //SYSUT1 DD //SYSLMOD DD SYSOUT=A DSN=&&ASMBL,DISP= (OLD/DELETE) UNIT=SYSDA,SPACE=(TRK,(20,5) ) DSN=SYS2.IDMS.OPPMON.SUBSDMCL(DMCL),DISP=OLD SYS2.IDMS.LOAD = data set name of IDMS load library SYSDDL = DD name of Data Directory SYSJRNL = DD name of Journal File SYSOUT=A = printed output files &&DMCL = data set name of temporary data set for DMCL output * = input unit &&ASMBL = data set name of temporary data set for assembler output DMCL = name for DMCL module (4) Subschema Compilation Functional Description The subschema, definition, is a logical subset of the schema database designed primarily to maintain program-data VI-7 ------- DMCJ. SOURCS ASSEMBLE UST1NG UNKEDIT MAP STEP 1 IOMSOMCL T STEP 2 ASSEMBLER STEP 3 LINKAGE EDITOR DEVICE-MEDIA INPUT USTING OBJECT DMCL MODULE FIGURE vi-4 DEFINING DiMCL MODULE ------- independence. Containing only those data elements to be referenced by the application program, the subschema definition remains largely independent of the schema or "global" definition. Consequently, changes may be made to a specific subschema without necessitating schema recompilation or extensive database reprogramming. The following points are important to an understanding of the relationship between the schema and subschema: The schema is established independently of any user program or any subschema. A user program invokes a subschema, i.e., sees the database in terms of a particular subschema's view. No user-written program can have access to an IDMS database without use of one (and only one) subschema view. Only the areas, records, data-items, and sets included in the subschema invoked by a program may be referenced by that program. Only the database management functions permitted within its subschema view are executable by the application program in the run-time environment. (For example, in terms of one particular subschema view, customer records may be OBTAINED but not ERASED.) The declaration of a subschema has no effect on the declaration of any other subschema, and subschemas may overlap one another in their 'views' of the database. The same subschema may be invoked by any number of programs. Any number of subschemas may be declared on the basis of a given schema Object versions of subschemas are compiled independently of any user program and stored in a 'load1 or 'core-image1 library. At least one user-established DMCL module must have been compiled into the IDMS Data Directory before the Subschema Compiler can be executed. The subschema compile phase will generate system control tables which describe a specific program view containing particular Areas VI-8 ------- Records (all or specified portions) Sets In addition, the subschema tables also reflect privacy lock restrictions limiting the programs using this view of the database to designated subsets of DML commands. No user program may request the services of IDMS at run- time without reference to a subschema. In such an execution- time environment no run-unit can successfully request access to a portion of the database not included in its subschema. It also cannot request a function for which it is not authorized according to that subschema's privacy locks. Figure VI-5 outlines the important steps comprising the subschema phase of IDMS data definition. These are described below: The Subschema Compiler is executed (1), accepting as input user-coded subschema descriptor statements, (2) defining a particular program view of the database. The Data Directory file is used as a resource to establish knowledge of the schema description of the database and ensure that each subschema DDL statement is consistent with that description. The presence of any area, data-item, record, or set included in the subschema is verified against the schema description. The compiler registers in the Directory all data pertaining to this particular subschema description, replacing any previous such information associated with the same subschema name. IDMSUBSC generates several output listings (3) during the course of its processing: The 'Subschema DDL Processor Input Listing1 displays all user-entered card images into the compile run, along with any diagnostics as appropriate. The 'Subschema Data Records' listing shows the make-up of each record type included in this subschema, as it would be visible to an application program. These data descriptions are in COBOL data division format. The 'Definition Entries' listing has three parts: Record - the names of all records included in this subschema view. These names are shown in COBOL data division format. VI-9 ------- SUBSCHEMA SOURCE STATEMENTS IDMSUBSC COMPILES SUBSCHEMA DDL INPUT USTING IDMS DATA DIRECTORY ASSEMBLES SOURCE ASSEMBLES USTINGIS) ^ V ASSEMBLES UNK EDIT MAP LINK EDIT * (3 ( OBJECT SUBSCHEMA MODULE FIGURE vi-5 SUBSCHEMA COMPILE PHASE ------- Set - the names of all sets included in this subschema view Area - the names of all areas so included A fourth type of listing, the 'DBMS Bind Calls,1 indicates all BIND statements (in COBOL CALL format) that an application program would have to issue in the run-time environment to make available all record types in this subschema view. Since the IDMS DML processors automatically provide these BIND source statements at compile time and since the call form these requests take is dependent upon compile mode, this listing is not referenced as frequently by the data administrator as it once was. The Subschema Compiler also produces an assembler source file (4) which is used as input to a standard OS or DOS assembly (5), which in turn is followed by a link edit (7). The listings coming out of these steps are the normal assembler listing(s) and link edit maps with which the user is familar. The object subschema module (9) which results is stored in the user core-image or load library. The name under which it is available must be the same as the 'subschema name1 assigned in the source subschema statements for the subschema view. The Subschema Compiler can be executed any number of times in conjunction with a given schema description. Thus, as many program views as the DBA thinks are necessary can be established. Each resultant subschema module can be shared by as many application programs as required. Syntax and Semantics The Subschema DDL (Data Definition Language) defines the user's view of the overall database. It consists of two major statement groups: SUBSCHEMA IDENTIFICATION DIVISION SUBSCHEMA DATA DIVISION The statement groups must appear in the sequence shown. Each describes a specific aspect of the subschema and begins with a descriptive paragraph name. VI-10 ------- The SUBSCHEMA IDENTIFICATION DIVISION statements identify the subschema name, Device-media, name, author, date, installation, and remarks. The SUBSCHEMA DATA DIVISION statements specify the area(s), record(s), and set(s) that are to be included in the subschema view being defined. Specific information concerning each Subschema Division statement can be found in the IDMS Database Design and Definition Guide, Section 8.1. Execution Instructions To define a subschema, execute the program IDMSUBSC followed by the assembler and linkage editor. Input to IDMSUBSC consists of a source deck written in subschema DDL. Output is as follows: A description of the subschema stored in the Data Directory An assembler source program A subschema DDL input listing, a subschema data records listing, and a definition entries listing Input to the assembler consists of the source program produced by the assembler. Output is as follows: A relocatable program An assembler listing Input to the linkage editor consists of the relocatable program produced by the assembler. Output is as follows: An object program A linkedit map Figure VI-5 illustrates these operations. The JCL for performing the operations is as follows: IDMSUBSC //STEP1 EXEC PGM=IDMSUBSC,REGION=130K //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSDDL DD DSN=SYS 2.IDMS.OPPMON.DIRECTRY,DISP= (OLD,PASS) VI-11 ------- //SYSJRNL //SYSLST //SYS OUT //SYSPCH //SYSIPT subschema //STEP2 //SYS PRINT //SYSLIB //SYSUT1 //SYSUT2 //SYSUT3 //SYS PCH //SYS IN //STEP3 //SYS PRINT //SYSLIN //SYSUT1 //SYSLMOD DD UNIT=TAPE62,VOL=OPPDIR,DSN=OPPJRNL,DISP=MOD DD SYSOUT=A,DCB=BLKSIZE=133 DD SYSOUT=A DD DSN=&&SUBSC,DISP=(NEW,PASS),UNIT=SYSDA, SPACE=(CYL,(5,1)), DCB=(RECFM=FB,LRECL=80,BLKSIZE=1500) DD * DDL source statements (omit if SYSIPT is other than *) EXEC PGM=IEUASM,PARM='NOLOAD,DECK',REGION=80K DD SYSOUT=A DD DSN=SYS1,MACLIB,DISP=SHR DD UNIT=SYSDA,SPACE = (CYL,(2,2)) DD UNIT=SYSDA,SPACE=(CYL,(2,2)) DD UNIT=SYSDA,SPACE=(CYL,(2,2)) DD DSN=&&ASMBL,UNIT=SYSDA,DISP= (NEW,PASS), SPACE=(80,(400,40)) DD DSN=&&SUBSC,DISP= (OLD,DELETE) EXEC PGM=IEWL,PARM='XREF,LIST,LET,SIZE = (96K,12K) REGION=96K DD SYSOUT=A DD DSN=&&ASMBL,DISP= (OLD,DELETE) DD UNIT=SYSDA,SPACE = (TRK, (20,5)) DD DSN=SYS2.IDMS.OPPMON.SUBSDMCL(SUBSC) ,DISP=OLD SYS2.IDMS.LOAD = data set name of IDMS load library SYSDDL = DD name of Data Directory SYSJRNL = DD name of Journal File SYSOUT=A = printed output &&SUBSC = name of temporary data set for subschema output * = input file &&ASMBL =name of temporary data set for assembler output SUBSC = name for subschema. (5) Data Base Procedure Functional Description Database procedures are routines associated with a specified DML function for a particular record type. Defined in the schema record description statements, these procedures intercept and process service requests between the application program and DBMS. Database procedures, which are normally written by the database administrator, are invisible to the application program. In general, database procedures increase the number of data management capabilities available to the DBA. Some specific procedure functions include the ability to: Modify a schema record on its way in and out of the database VI-12 ------- Compress/expand data-items and records to save disk storage space Validate the data-items and establish data integrity Encode/decode data-items and records to ensure physical data security Compute data-items based upon the value of other data- items Perform search algorithms and gather statistics Ensure data privacy at the record occurrence level as described below: An important feature concerning database procedures is that procedures have access to the schema record occurrence, while the application program, of course, has access only to the segmented subschema record. This fact allows the DBA to implement privacy locks within database procedures that will intercept and protect sensitive data (i.e., those few occurrences of a given record type that may not be viewed by an application). For example, the DBA may write a procedure to allow a program access to all personnel records except those with account numbers less than 1000. If the record fails the procedure's range check, access to the record is denied, and a user error code is returned to the program, indicating violation of a DBA-implemented privacy lock. Thus, database procedures can determine not only which record types may be seen by an application program, but also which occurrences of a given type may be accessed. The end result is that the DBA can provide record occurrence privacy through database procedures. Service application database requests from non-IDMS tape or disk files. Using procedures, the DBA can include non-IDMS files in his configuration and process application database requests from those files. The application program remains unaware of either the procedure's presence or the source of the logical records returned to working storage. Normal application processing continues despite the switch from IDMS to non-IDMS mode. The application program still sees only the segmented subschema record. Thus, the advantages of data independence are extended to non-IDMS records and files. VI-13 ------- Syntax and Semantics A database procedure, which is defined in the schema record description statements and associated with one or more DML functions, is invoked using the following CALL command: CONNECT BEFORE MODIFY CALL database-procedure-name AFTER DISCONNECT ON ERROR DURING ERASE FIND GET This statement is formatted according to the following rules: Multiple procedures may be associated with the same record type by specifying separate CALL clauses for each procedure. However, the following options apply to any procedure: One or more DML entrance times (BEFORE, ON ERROR DURING, AFTER) may be specified for the same procedure. One or more DML functions may be specified for the same procedure. The same procedure may be specified in different CALL clauses, and thus may be associated with different record types. Procedure entrance time is controlled by specifying one of the following options: BEFORE - the procedure is entered immediately prior to the performance of the DML function(s). AFTER - the procedure is entered immediately after the DML function and error procedures, if any, have been performed. ON ERROR DURING - this is a special form of AFTER procedure which is entered each time DBMS detects an error condition (i.e., error status indicator not equal to zeros) during the performance of the DML function. Procedures are executed only on explicit DML functions. For example, if there is an implied CONNECT on a STORE (set membership is automatic), a procedure declared as CALL . . . BEFORE CONNECT would not be invoked at the time a STORE is executed. VI-14 ------- Coding Data Base Procedures A Data Base Procedure is coded as a subroutine employing standard OS calling sequences. It is invoked by IDMS on behalf of an application program. When invoked, program control is passed along with the following information blocks: Procedure Control Block Application Control Block Application Procedure Communication data Record Control Block User Record Occurrence The Procedure Control Block provides information reflecting the general conditions under which the procedure is being invoked. The information passed is the same for any procedure invoked by any DML function. The Application Control Block is a second fixed block of information that reflects conditions in the application program at procedure execution time. This data includes the name of the subschema associated with the application program that caused the procedure to be invoked followed by a copy of the first 200 bytes of the user IDMS Communcations Block. This is the first portion of the same area used for all communication between IDMS and application programs. The third block of information passed by IDMS is the Application-Procedure Communication data. The contents of this block are determined completely by the user and include any data to be passed between the procedure and the application. Consequently, the area may consist of any number of user-defined data fields. Normally, however, the procedure will operate invisible to the application program. In these instances, the null indicator is set by IDMS to an alphanumeric value of 'NULL1, indicating that no information is being passed between the application and procedure. The Record Control Block gives the procedure information regarding the specific record type upon which it is operating. This information, which consists of data specific to record-level procedures, is essential, since any procedure may be invoked for more than one record type. The fifth argument passed by IDMS is the actual record occurrence as described in the schema, including all data- items within the record regardless of the application's subschema view. A detailed description of the layout and contents of these blocks is found in the IDMS Database Design and Definition VI-15 ------- Guide, 10.5-10-10. Link-Editing Database Procedures Data-base Procedures are compiled, link-edited and stored into a load library which contains all IDMS data base procedures. VI-16 ------- 2. APPLICATION PROGRAM SUPPORT (1) Data Directory Reports Report Description An integral part of the IDMS system is a group of ten Data Directory reports provided by the IDMSRPTS program. These reports furnish system documentation for use by programmers, analysts, database administration staff, data processing management, and user department personnel. Figure VI-6 illustrates the reports available from the Data Directory. The reports and brief descriptions are listed below: Range rtap - ranges of each area within the schema and each record type within each area Device-Media Control Listing - physical file and corresponding area mapping information as defined in the schema Program Cross References - each host language program (COBOL or PL/I) written under each subschema, including the most recent compilation date Schema/Subschema Record Descriptions - schema and subschema-view record specifications Schema/Subschema Set Descriptions - schema and program- view set specifications Subschema Data Dictionary Listing - specifications for records and all component data items, particularly for use by non-EDP personnel in conjunction with the IDMS/CULPRIT retrieval system Procedure/COPY Listing - record descriptions and procedure statements categorized by language or by mode Protocol Listing - specifications of user environment 'calling sequences' and formats. Several of the Data Directory reports are designed for use by management and user department personnel, as well as programmers and DBAs. In general, the Data Directory Reports Program (IDMSRPTS) communicates all database specifications and constraints to the user. VI-17 ------- REPORT PARAMETERS DATA DIRECTORY REPORT PROGRAM DATA DIRECTOR? FILE SCHEMA SETS (SE70ES) 3POGRAM CROSS-REF (PGMLSTj ATS D I RECTOR? •_ISTING {OATQIRj FIGURE vi-6 OQCUMENTATION FROM THE DATA DIRECTORY SYSTEM ------- Input Parameters To provide maximum flexibility in selecting reports, the IDMSRPTS control parameter syntax permits the user to request reports for any number of schemas and subschemas. IDMSRPTS syntax follow: SCHEMA = (schema-name-1 [schema-name-2] . . .]_ HIGHEST VERSION = ALL (schema-version-1 [schema-version-2] . . .]_ SUBSCHEMA = ALL (subschema-name-1 [subschema-name-2] . . .)_ ALLRPT RNGMAP DMCLST PGMLST RECDES SETDES SUBREC SUBSET DAIDIK MODLST PRTLST Specific rules for each statement can be found in the IDMS Database Design and Definition Manual Section 9. JCL To generate Data Directory reports, execute the program IDMSRPTS. Input to IDMSRPTS consists of parameter cards and the information contained in the Data Directory. Output can be any of the following as requested on the parameter cards: Range map (RNGMAP) Device-media control listing (DMCLST) Program cross-reference listing (PGMLST) Schema record descriptions (RECDES) Schema set descriptions (SETDES) Module/copy listing (MODLST) Protocol listing (PRTLST) VI-13 ------- Subschema record descriptions (SUBREC) Subschema set descriptions (SUBSET) Data dictionary listing. (DATDIR) The JCL for performing the operation is as follows: IDMSRPTS //STEPLIB //SYSDDL //SYSJRNL //SYSOUT //SYSLST //SYSIPT parameter cards EXEC PGM=IDMSRPTS,REGION=180K DD DSN=SYS2.IDMS.LOAD,DISP=SHR DD DSN=SYS2.IDMS.OPPMON.DIRECTRY,DISP=(OLD,PASS) DD DUMMY DD SYSOUT=A DD SYSOUT=A,DCB=BLKSIZE=133 DD * SYSDDL = DD name of Data Directory SYSJRNL = DD name of Journal File SYSOUT=A = printed output (2) DML Protocols Application Program Usage of Protocols The use of various DML protocols controls the translation of DML statements into the appropriate programming language source code. The first level of control is the programming language itself, that is identification of the correct syntax for this program. The second function is to vary the translation of the DML statements as it applies to various modes of operation (e.g., batch vs teleprocessing). The third major function is to identify programs into which the DML processor is to copy additional data descriptions or code sections. Utility Description The IDMS Data Directory Utility (IDMSCLUC) establishes, updates, and deletes various controls for the DML processors. Specifically, IDMSCLUC permits the user to: Build non-IDMS data structures into the Dictionary/Directory file for subsequent COPY by the DMLC or DMLP processors Store and COPY module source code (e.g., IDMS-STATUS SECTION) VI-19 ------- Establish protocol information needed by the DMLC and DMLP processors in generating "calling sequences' between application programs and IDMS Register programs in the Directory file to restrict program compilation against specific subschemas Delete schema descriptions from the Directory Transfer program activity statistics between subschema and schema versions Establish a system-wide quote character. A Full Description of the IDMSCLUC control options along with information on individual action codes can be found in the IDMS Database Design and Definition Manual, Section 12. Utility Execution Prior to executing the DML processors, the IDMSCLUC utility must be executed. Input to IDMSCLUC consists of a source deck of CLUC statements. These CLUC statements can be found on the IDMS, version 5.0 installation tape in file 3 (SOURCE), member = IDMSPROT. Input only the source needed by OPP (COBOL, BATCH). Output is as follows: Protocol and other information stored in the Data Directory A CLUC source listing. Figure VI-7 illustrates this operation. The JCL for performing this operation is as follows: IDMSCLUC // EXEC PGM=IDMSCLUC,REGION=180K //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSDDL DD DSN=SYS 2.IDMS.OPPMON.DIRECTRY,DISP= (OLD,PASS) //SYSJRNL DD UNIT^TAPE 62,VOL=OPPDIR,DSN=OPPJRNL,DISP=MOD //SYSOUT DD SYSOUT=A //SYSLST DD SYSOUT=A //SYSIPT DD * CLUC source statements (omit if SYSIPT is other than *) SYSDDL = DD name of Data Directory SYSJRNL = DD name of Journal File SYSOUT=A = printed output VI-20 ------- CLUC CONTROL STATEMENTS IOMSCLUC DATA DIRECTORY SOURCE UST1NG HGURE VI-7STORING PROTOCOL AND OTHER INFORMATION ------- (3) Program Compilation and Link-Edit Procedures JCL for COBOL Program COBOL programs are preprocessed by the DMLC compiler to convert DML statements into standard COBOL. Output is as follows: A source COBOL program A DML source listing and diagnostics Input to the ANS COBOL compiler consists of the source program produced by IDMSDMLC. Output is as follows: A relocatable program COBOL listings Input to the linkage editor consists of the relocatable program. Output is as follows: An object program A link-edit map Figure VI-8 illustrates this operation. The JCL to perform the preprocessing and compilation is as follows: //STEP1 EXEC PGM=IDMSDMLC,REGION=160K //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSDDL DD DSN=SYS2.IDMS.OPPMON.DIRECTRY,DISP= (OLD, PASS) //SYSJRNL DD UNIT=TAPE62,VOL=OPPDIR,DSN=OPPJRNL,DISP=MOD //SYSLST DD SYSOUT=A,DCB=BLKSIZE=133 //SYSPCH DD DSN=&&DMLC,DISP=(NEW,PASS),UNIT=SYSDA, // SPACE=(TRK,(5,5)), // DCB = (RECFM=FB,BLKSIZE=3120,LRECL=80) source statements (omit if SYSIPT is other than *) EXEC PGM=IKFCBLOO,REGION=160K PARM='DECK,NOLOAD,NOLIB,BUF=50000,SIZE=150K' SYSOUT=A DSN=SYS1.COBLIB,DISP=SHR UNIT=S YSDA,SPACE = (TRK, (10, 5) ) UNIT=SYSDA,SPACE=(TRK, (10,5)) UNIT=SYSDA,SPACE=(TRK, (10,5)) UNIT=SYSDA,SPACE=(TRK, (10,5)) DSN=&&COB,DISP=(NEW,PASS),UNIT=SYSDA, //SYSIPT COBOL/DMLC //STEP //SYS PR I NT //SYS LIB //SYSUT1 //SYSUT2 //SYSUT3 //SYSUT4 //SYS PUNCH DD source EX) DD DD DD DD DD DD DD VI-21 ------- SOURCE PROGRAM COBOL LISTINGS £r DIAGNOSTICS UNKMAP STEP 1 IOMSDMLC STe? 2 ANS COBOL COMPILER STE? 3 LINKAGE EDITOR OPTIONAL SOURCE LISTING DIAGNOSTICS OBJECT PROGRAM FIGURE vi-SCOMPILJNG COBOL PROGRAM IOMS 1ISE3 PROGRAM RUN-UNIT FIGURE VI~9 EXECUTING A PROGR.-O1 IN LOCAL MODE ------- // SPACE=(80,(400,40)) //SYS IN DD DSN=&&DMLC,DISP= (OLD, DELETE) //STEP3 EXEC PGM=IEWL,PARM='XREF,LIST,LET,SIZE = (95K, 12K) // REGION=96K //SYSUT1 DD UNIT=SYSDA,SPACE=(TRK,(20,5)) //LIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYSLMOD DD DSN=SYS2.IDMS.OPPMON.PROG. (USERPROG ) ,DISP=OLD //SYSLIN DD DSN=&&COB,DISP=(OLD,DELETE) // DD * /INCLUDE LIB (IDMS,IDMSCANC) //SYSLIB DD DSN=SYS1.COBLIB.DISP=SHR SYSDDL = DD name of Data Directory SYSJRNL = DD name of Journal File SYSOUT=A = printed output &&DMLC = name of temporary data set for DMLC output SYSl.COBLIB = data set name of ANS COBOL Library &&COB = name of temporary data set for COBOL output SYS2.IDMS.OPPMON.PROG (USERPROG) = name of user applications program (4) Execution Instructions In local mode, the applications program is linked to IDMS, the batch interface module. The program is started as a job step and the interface module loads the appropriate subschema and a copy of the DBMS. The DBMS then loads the appropriate DMCL module. Figure VI-9 illustrates database access via a local DBMS. The JCL to start a program using IDMS in local mode is as follows: LOCAL MODE //STEPLIB //IDMSJRNL //DATABASE //CODEFILE //INDEX //IDMSJRNL 9 //SYS OUT //SYSUDUMP USERPROG IDMS. LOAD EXEC PGM=USERPROG DD DSN=SYS2.IDMS.LOAD,DISP=SHR DD DSN=SYS2.IDMS.OPPMON.PROG,DISP=SHR DD DSN=SYS2.DMS.OPPMON.SUBSDMCL,DISP=SHR DD DSN=SYS2.IDMS.OPPMON.DBRROC,DISP=SHR DD DSN=IDMS.JRNLFILE,DISP=MOD,UNIT=TAPE62, VOL=PRIVATE DD DSN=DBFILE1,DISP=OLD DD DNS=DBFILE2,DISP=OLD DD DSN=DBFILES3,DISP=OLD DD DSN=IDMS.JRNLFILE,DISP=MOD,UNIT=TAPE62, VOL=PRIVATE additional JCL for applications program, as required DD SYSOUT=A DD SYSOUT=A name of user's applications program ! data set name of IDMS load library VI-22 ------- IDMS.JRNLFILE = data set name of Journal File DBFILE1 = data set name of database DBFILE2 = data set name of code file DBFILE3 = data set name of index file SYSOUT = A = printed output VI-23 ------- 3. DATA BASE LOADING (1) Data Base Initialization File Allocation The OPP Monitoring Data Base will be allocated as three OS files. The largest file contains the main data base and encompasses the STATE area and the RESIDUE area. The file will be allocated initially to be 70% full. The second file contains codes found in the data base along with their associated meanings. This file requires 3 data base pages or 1 track. The third file holds the indices for the sequential processing facility. Prior to allocating this file, the utility described in the Data Base Design volume (page 52) must be run to obtain the size parameters. The following JCL will be used for the allocation. // EXEC PGM=IEFBR14 //DATABASE DD DSN=DBFILE1,UNIT=3 330-1,VOL=SER=USRVOL // SPACE=(3156,20575),DISP=(NEW,CATLG) //CODEFILE DD DSN=DBFILE2,UNIT=3 330-1,VOL=SER=USRVOL // SPACE = (3156,4),DISP=(NEW CATLG) //INDEX DD DSN=DBFILE3,UNIT=3330-1,VOL=SER=USRVOL, // SPACE=(1180,numpg) ,DISP= (NEW,CATLG) DBFILE1 = data set name for DATABASE file DBFILE2 = data set name for CODEFILE file DBFILE3 = data set name for INDEX file Area Initialization IDMSINIT must be the first program run against any IDMS database prior to processing. This step is mandatory, since each page of the user's database must be formatted prior to storing records in the database. Consequently, IDMSINIT performs the following initializing actions: Writes out direct access blocks (one block = one page) to the database Formats each page (e.g., writes page numbers) Reformats all or sections of the database following device (e.g., head crash) or program failure Depending on user specification, IDMSINIT initializes the database in its entirety or by FILE. In addition, IDMSINIT may be used to re-initialize the database by AREA or FILE. VI-24 ------- In order to initialize the data base in its entirety the following cards must be used: PROCESS = TOTAL,DMCL=OPPDMCL To initialize or reinitialize the database, execute the program^IDMSINIT. Input consists of parameter cards. Output is as follows: Formatted or reformatted pages in the database A parameter listing Figure VI-10 illustrates this operation. The JCL for performing the operation is as follows: IDMSINIT // EXEC PGM=IDMSINIT //STEPLIB DD //SYSLST DD //DATABASE DD //CODEFILE DD //INDEX DD //SYSIPT DD parameter cards DSN=SYS2.IDMS.LOAD,DISP=SHR SYSOUT=A DSN=DBFILE1,DISP=OLD DSN=DBFILE2,DISP=OLD DSN=DBFILE3,DISP=OLD SYSOUT=A = printed output DBFILE1 = data set name of database file DBFILE2 = data set name of code file DBFILE3 = data set name of index file (2) Initialization of CODE Area Prior to loading the main database, the CODE record must be initialized and loaded. An application program (CODEINIT) must be run which will store blanks in the CODE field and 20 in the length field. Then a second application program (CODELOAD) must be run which will input a file containing the CODE, NAME combinations and store the values into the database CODE record. (3) Data Base Loading Loading of the main data base requires running two programs. The first program (RESLOAD) will input a file containing the residue codes, residue class, names, and system warning levels and store these values into the data base RESIDUE record. The second program (DBLOAD) will input files containing information on states, sites, and samples and will load the rest of the data base for all monitoring systems. The input files should be sorted into state, site, sample date order prior to running DBLOAD. VI-25 ------- PARAMETER CARO(S) IOMSINIT DATABASE PARAMETER UST1NG FIGURE VI-10 INITIALIZING THE DATABASE ------- 4. DATA BASE UPDATING (1) Input File Creation The OPP Monitoring Data Base will be updated via input transactions processed by the Edit/Update Subsystem. Three input files can be used depending on the type of Edit/Update being run. These files are as follows: "New" input transactions Previously rejected Humans transactions Previously rejected non-Humans transactions. The latter two files are created as output by a previous run of the edit/update subsystem. These two files will initially be empty the first time the subsystem is run and also when no transactions are rejected during the previous run. The new input transaction file is created by keying the input forms provided to SSB by FMB. (2) Edit/Update Cycle The users are to provide the type of cycle to be run. Two modes are possible. These are an edit only mode where transactions are edited with no attempt to update, and edit and update mode where transactions are edited and if clean an attempt to update is made. The mode must be provided as input to the subsystem by the programmer responsible for system operation. In either mode of operation, reject files are created. Additional output include an error list providing details of errors in rejected transactions. If in the edit only mode, all transactions are rejected to the reject files. (3) Correction Cycles The error list is sorted by transport mode and should be referred to EMB with the input forms. The Project Officer will then make the required corrections to the transactions by accessing the appropriate reject files via ALPHA. The next time the update program is run, the new rejection files are used as input along with any new input transaction file created since the last update run. VI-26 ------- 5. BACK-UP AND RECOVERY (1) Database Back-Up Backup simply means keeping a second copy of the database. Two backup media are provided under IDMS: Dump Pile By dumping the database, the user obtains a complete picture of the database as it existed at a point in time. Journal File By recording checkpoints and data updates as they are made, the user maintains a running picture of the database. Database Dump Description The purpose of the IDMSDUMP program is two-fold: To copy all or specified AREAS of an IDMS database to tape for backup security To provide a database statistics report IDMSDUMP performs the following functions: IDMSDUMP reads a PROCESS card to determine the user's run time requirements. If PROCESS=AREA is specified, additional cards follow the first, giving the names of the AREAS to be copied. If REPORT=YES or ONLY is specified, IDMSDUMP will collect data and compile reports about the current content and status of the database, including space utilization and record distribution data. If PROCESS=AREA is designated the reports are generated by AREA; for PROCESS=TOTAL the report covers all AREAS known to the DMCL tables entered. Database Dump Syntax First Card YES PROCESS = T2IJL, DMCL=device-media-name , REPORTS=NO VI-27 ------- Subsequent Cards . . . AREA=area-name Specific information on each statement can be found in the Database Design & Definition manual, Section 11-34. IDMSDUMP Output Format IDMSDUMP writes the dump tape in a fixed format such that block sizes are equal to the page size of the database. In the first block of the file, the program writes the following information only: Date the dump is taken - MM/DD/YY - 8 bytes Names of AREAS dumped - 16 bytes The literal 'END1 to specify end of AREA names. One or more blocks may be written, depending on the number of AREA names which must be stored. Following these blocks, the database is written beginning with the lowest numbered page to be copied. Data is dumped in ascending order by logical page number. IDMSDUMP optionally produces a Database Utilization Report consisting of the following three sections: Section I is a summary report of the area being dumped and contains the following information: Gross page size Number of pages dumped AREAS dumped (as known to the DMCL tables) Space used (i.e., number of characters occupied) Space remaining (i.e., number of characters of storage available) File utilization expressed as a percentage of space used vs. space available rounded up to the next integer. Section II is a space distribution table, indicating how the occupied space is allocated among each record type. The table, which can keep track of up to 1000 record types, contains the following information: VI-28 ------- Record type listed by record ID assigned in the schema Number of occurrences of each record type found in the dumped area Total space used - length multiplied by number of occurrences Percentage of space used - the percentage of the total space occupied by this record type Section III is a second space distribution table, indicating: Occupancy range - the number of characters per page in multiples of 100 through 1,599 Number of pages in each range. If PROCESS=TOTAL is specified on the IDMSDUMP input PROCESS card, the utility produces area as well as global total reports. Database Dump JCL Figure VI-11 illustrates the operation. Use the following JCL to run IDMSDUMP: // EXEC PGM=IDMSDUMP //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYS001 DD DSN=IDMS.DUMP,DISP=(NEW,KEEP),UNIT=TAPE62, // DCB=BLKSIZE=3156 //DATABASE DD DSN=DBFILE1,DISP=OLD //CODEFILE DD DSN=DBFILE2,DISP=OLD //INDEX DD DSN=DBFILE3,DISP=OLD //SYSLST DD SYSOUT=A //SYSIPT DD * parameter cards IDMS.DUMP = data set name of dump tape DBFILE1 = data set name of data base DBFILE2 = data set name of CODE file DBFILE3 = Data Set Name of INDEX file SYSOUT=A = printed output Database Dump Frequency The frequency of data base dumps is a variable of the amount of updating and the required time for recovery in the event of a system crash. Since the OPP Monitoring system is batch and the level of data base updates is relatively low, a weekly dump is recommended. VI-29 ------- PARAMETER CARO(S) IDMSDUMP DATABASE FIGURE v-i-llDUMPING THE DATABASE ------- Journaling The journal tape provides a running record of all data base changes which have taken place since the last data base dump was taken. A new tape (or series of tapes) should be started following each back-up data base dump. The combined use of these two sets of tapes for recovery purposes will be discussed in the next section. The journal tape must have a standard OS tape label. If the IDMS data base sustains heavy usage, a dedicated tape drive is desirable. Oherwise, the journal tape must be kept in the operations area and remounted each time an IDMS job runs. The IDMS journal file contains each run-unit's identity, starting time, ending time, and any database changes effected during program execution. As each program signs onto the DBMS, a beginning-of-the-job checkpoint is written to the journal tape. The checkpoint contains various program-id, starting time, and area usage mode information. If a program READIES an area in an UPDATE usage mode, 'before1 and 'after1 page images are written to the journal for each page changed. 'Before1 images provide a 'snapshot1 of a database page prior to modification(s). 'After' images provide a page image that includes all changes made while the page was in the buffer pool. When a run-unit issues a FINISH for a given area, an end-of-job checkpoint is logged on the journal. This checkpoint contains program-id and ending time information as well as run-unit statistics. These statistics include the number of pages read from and written to the database, as well as other information. If the run-unit terminates abnormally, an ABRT checkpoint that does not contain these access statistics is written to the journal file. The DBA may use the contents of the journal tape in conjunction with the IDMSJFIX, IDMSRBCK, and IDMSRFWD utilities to recover the database following a system failure. The utilities will restore the database as follows: IDMS Rollback erases a run-unit's effect by reapplying 'before1 page images to a specified BGIN checkpoint. IDMS Rollforward restores the database to a specified time by applying 'after1 images to all or selected database areas. A Journal File is created by assigning a file to an application program (local mode) or CAMP (Central Version). IDMS automatically writes the following checkpoints and data images on the Journal File: VI-30 ------- BGIN Checkpoint - Written when an application program becomes active. An AREA checkpoint is sometimes used for overflow information from a BGIN checkpoint. ENDJ Checkpoint - Written when a program terminates normally. ABRT Checkpoint - Written when program accessing the database via CAMP terminates abnormally. An ABRT checkpoint cannot be written for a local program that terminates abnormally. DATA Before Image - Written when a DML command is issued that will change data in the database. The affected database page is written to the Journal File before the change is made. DATA After Image - Written when data is changed in the database. The affected database page is written to the Journal File after the change is made, but before the page is written to the database. (2) Recovery Recovery Techniques When a hardware or software failure causes a program(s) using the database to fail or causes data on the disk to be physically lost, a recovery operation is necessary. The procedures required for the recovery depend on the nature of the crash: Program Running in Local Mode Fails - Before the program can be restarted, all changes made to the database by the aborted program must be nullified; i.e., the data as it existed at the start of the program must be restored. System Fails - If the operating system fails, it is necessary first to close out the Journal File properly: ABRT checkpoints for the programs that were active at the time of the crash and an end-of-file marker must be written. The database can then be restored to a point where no programs are operating in update mode and all affected programs can be restarted. Disk Fails - On a disk failure, data is physically lost. A new database must be initialized and loaded using the latest Dump File. The Journal File tapes are then used to restore the database to the latest point where no programs are running in update mode. Programs running at the time of the crash can then be restarted. VI-31 ------- Each of the above causes of a system crash are explained in detail in the following paragraphs. Figure VI-12 illustrates the crash and recovery of a local applications program. The steps involved after the crash is detected are as follows: Execute the Program IDMSRBCK with the operation PROCESS=TOTAL. If more than one program has used the Journal File, specify the date and time of the last (most recent) BGIN checkpoint. Restart the applications program. Usually it is best to use a new file for the Journal File. When a program running under the Central Version fails, a console message appears indicating that either: The program will be recovered automatically. This is possible if no programs are operating in UPDATE mode. The program cannot be recovered automatically due to a usage mode conflict. Figure VI-13 illustrates the recovery of the database after a system crash. The steps involved are as follows: Recover the operating system. Execute the program IDMSJFIX. This produces (1) a new Journal File with ABRT checkpoints for the programs that were active at the time of the crash, plus an EOT marker, and (2) PROCESS=TOTAL and date/time cards for IDMSRBCK. Execute the program IDMSRBCK using the new Journal File and the parameter cards produced by IDMSJFIX. IDMSRBCK will recover the database to the most recent quiesce level-0 point. Restart all programs affected by the IDMSRBCK operation. Figure VI-14 illustrates the recovery of the database after data is physically lost due to a disk failure. The steps involved after the problem is detected are as follows: Abort local applications programs Initialize all or the affected part of the database on a new disk(s) with the program IDMSINIT. VI-32 ------- / / SGIN PGM-A 3EFORE INV « 2022 PGM-A AFTER INV = 2073 PGM-A ; ( ( I 1 BEFORE INV = 2073 PGM -A AFTER INV = 2096 PGM-A ( ) 1 \ 3EFORE INV = 2096 PGM-A AFTER INV - 3001 PGM-A ( j DATABASE FIGURE VI-12EXAMPLE OF RECOVERING DATABASE FOR LOCAL PROGRAM. ------- BGIN PGM-A 3EFORE INV * 2022 PGM-A AFTER INV « 2073 PGM-A 3G1N PGM-3 BEFORE INV • 2073 PGM-S AFTcR INV - 2063 PGM-3 BEFORE I.NV » 2063 PGM -A AFTER INV - 2086 PGM-A A8RT ?GM-A A3RT PGM -3 OATASASe FIGURE VI-13 EXAMPLE OF RECOVERING DATABASE AFTER SYSTEM FAILURE. ------- IOMSHFWD USAGE REPORT IOMSINIT IDMSHSTR IOMSRFWD WOSTRECEMTCUIESCS LEVEL -0 POINT FIGURE VI-LI EXAMPLE OF RECOVERING DATABASE AFTER DISK FAILURE. ------- Restore the database from the most recent Dump File to the new disk(s) using the utility IDMSRSTR Using the most recent Journal File tape, execute the utility IDMSRFWD with the option PROCESS=REPORTS Determine from the usage report produced by IDMSRFWD the date and time of the most recent 3GIN checkpoint where the quiesce level is 0 - i.e., the most recent point where no programs were running in update mode Using all Journal Files taken since the dump, execute the utility IDMSRFWD with the options PROCESS=TOTAL, and RESTORE=YES, specifying the date and time of the most recent quiesce level-0 point. Restart all programs that began after the most recent quiesce level-0 point. Journal Tape Maintenance IDMSJFIX (Journal Fix) is used to close properly a journal tape that does not contain a normal end-of-file mark. This abnormal condition can occur for several reasons - hardware crash, power failure, or operating system malfunction. Program functions are as follows: IDMSJFIX reads a syntax card to identify the user's DMCL tables The utility is a tape-to-tape copy program that reads from external file name SYS001 (input) and writes to external file name SYS002 (output). IDMSJFIX continues to read and write tape blocks until it encounters a wrong block length error or a date and time earlier than the date and time of the previous block written. When neither of these conditions occurs, IDMSJFIX assumes that it has reached end-of-file. At end-of-file IDMSJFIX: Writes ABRT (abort) checkpoints for all active run-units Prints a parameter and checkpoint listing (see "Report Format1 below) Generates two parameter cards to be used as input to IDMSRBCK: PROCESS card VI-33 ------- RESTORE, PRINT, DATE, and TIME parameter card Under OS, sets a condition code indicating whether or not the recovery cards were generated Prints a message stating whether or not database recovery is necessary Generates cards to rollback database to last quiescent point. Writes a normal end-of-file mark to close out the journal tape. In order to generate an ABRT checkpoint and create the punched card input required by IDMSRBCK to roll back the database to a quiesce point, IDMSJFIX must have access to the beginning-of-job checkpoint. This requirement becomes especially important if a long-running batch update program (with more than one reel of journal file output) were operating when the system went down. In this instance, the user must begin execution of the IDMSJFIX utility with the journal file positioned to the beginning of tape for the first volume (the one with the beginning-of-job checkpoint on it). The Journal Fix program will copy all reels of the file, despite the fact that intervening volumes do not require an EOF mark. To write a proper Journal File when the original Journal File does not terminate at normal end-of-file, execute the program IDMSJFIX. Figure VI-15 illustrates the operation. The JCL to run the above operation is: //STEPLIB //SYS 001 // //SYS 002 DD //SYSOUT DD //SYSIN DD parameter cards PGM=IDMSJFIX DSN=SYS2.IDMS.LOAD,DISP=SHR DSN=IDMS.JRNFILE,DISP= (OLD,PASS),UNIT=TAPE62, VOL=PRIVATE,DCB=BLKSIZE=3200 DSN=IDMS.JRNLFILE,DISP= (NEW,PASS),UNIT=TAPE62, VOL=PRIVATE,DCB=BLKSIZE=3200 SYSOUT=A IDMS.JRNLFILE = data set name of bad Journal File (SYS001) and fixed Journal File (SYS002) 3200 = size of database page + 44 bytes SYSOUT=A = printed output Database Restore The IDMS Security Restore Program (IDMSRSTR) uses the data contained on a dump tape produced by the IDMSDUMP program to restore all or specified AREAS of the database. VI-34 ------- PARAMETER CARDS BAD \ F1L£ I IOMSJFIX f FIXED V FILE V < USTING TIGURL Vl-Ib FIXING A JOURNAL FILE ------- Program operation can be described as follows: IDMSRSTR reads a PROCESS card to determine the type of restore required and the name of the device-media control (DMCL) tables for the run. If PROCESS=TOTAL is specified, no additional cards are required. All AREAS represented on the input dump tape (and known to the DMCL tables) will be restored. If PROCESS=AREA, additional cards follow the first to specify the AREAS to be restored. Initialize must be executed if restore will reference a different (new) portion. To restore all or portions of the database from a dump tape, execute the program IDMSRSTR. Figure VI-16 illustrates the operation. Use the following JCL to run IDMSRSTR: IDMSRSTR // EXEC IDMSRSTR //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR //SYS001 DD DSN=IDMS.DUMP,DISP=(OLD,PASS),UNIT=TAPE62, // DCB=BLKSIZE=3156 //DATABASE DD DSN=DBFILE1,DISP=OLD //CODEFILE DD DSN=DBFILE2,DISP=OLD //INDEX DD DSN=DBFILE3,DISP=OLD //SYSLST DD SYSO(JT=A //SYSIPT DD * parameter cards SYS001 = DD name ofS-dump tape IDMS.DUMP = data set name of dumpt tape TAPE62 = symbolic device name of dump tape 3156 = tape block size; must be multiple of page size DBFILE1 = data set name of database DBFILE2 = data set name of CODE file DBILFE3 = dta set name of INDEX file SYSOUT=A = printed output Roll-Forward IDMSRFWD is a diagnostic and recovery tool that performs the following tasks: Prints page images from the journal file Restores 'after' images to the database VI-35 ------- PARAMETER CARO(S) IDMSRSTH 0ATA3AS£ FIGURZ VI-16 RESTORING THE DATABASE FROM A DUMP TAPE PARAMETER CARD(S) IDMSRFWO DATABASE U STING FIGURE VI-17 RESTORING .AFTER IMAGES ------- Prints a run-unit/area usage report These three functions may be specified independently of one another. It should be noted that unlike IDMSRBCK, which restores the database from its final to its initial (prior to processing) status, IDMSRFWD builds up the database from initial to a determined status with respect to all changes contained on the journal file(s). Program functions are: IDMSRFWD reads a PROCESS card to determine whether processing will be TOTAL or by AREA. TOTAL indicates that 'after1 images will be applied to the entire database to a specified point in time. AREA indicates that 'after' images will be applied only to pages within the AREA specified on the parameter card. IDMSRFWD reads the journal tape forward and: Optionally rebuilds the database (RESTORE=YES or RESTORE=NO); the utility will restore the database in whole or in part, according to whether PROCESS=TOTAL or PROCESS=AREA is specified Optionally prints 'before1 and 'after' page images depending on whether or not the PRINT option is specified Optionally prints the run-unit/area usage report only, without restoring the database If the PRINT option is specified (i.e., PRINT=BOTH, DECIMAL, or HEXADECIMAL), the following information is printed: Program name and unique main memory address on each 'before1 and 'after' page image Hexadecimal page images (PRINT=BOTH or PRINT=HEXADECIMAL) with page headers and footers Decimal page images (PRINT=BOTH or PRINT=DECIMAL) with page headers and footers and a single line documenting the type of change and line number of the record last changed on the page Even if the PRINT option is specified as NO, IDMSRFWD always produces: Parameter listing VI-36 ------- A single-line job checkpoint listing End-of-job statistics for each run-unit Finally, since the utility processes the journal one volume at a time, the user must ensure that the volumes are supplied in the correct order. IDMSRFWD maintains a 'quiesce level1 counter for each program on the journal tape. The counter mechanism works as follows: A BGIN (beginning-of-job) checkpoint in any type of UPDATE usage-mode increments the counter by one. An ENDJ (end-of-job) or ABRT (abort) checkpoint in any type of UPDATE usage-mode decrements the counter by one. A counter level of zero indicates that the database is in a quiesce (non-UPDATE) state at that point. Normally the user restores the database to a designated quiesce point (i.e., a point at which no update run units were in execution). However, it is possible to roll forward to a non-quiesce point (level number greater than zero), but the user should be aware that one or more programs were at that point potentially changing the database. Thus, there is no guarantee of 'logical integrity1 in the database if IDMSRFWD is executed to a non-zero checkpoint. 'Physical integrity,1 the assurance that there are no broken chains or related structural problems in the database, is guaranteed whenever the database is rolled forward to any IDMS-created journal checkpoint (quiesce or non-quiesce). This is true since all system buffers are written to the journal file and the database whenever a run-unit issues a FINISH command. To restore after images from the Journal File to the database, print page images from the Journal File, and/or print a usage report, execute the program IDMSRFWD. Figure VI-17 illustrates the operation. Use the following JCL to run IDMSRFWD: IDMSRFWD // EXEC PGM=IDMSRFWD //STEPLIB DD DSN=SYS2.IDMS.LOAD,DSP=SHR //SYS 009 DD DSN=IDMS.JRNLFILE,DISP=(OLD,PASS),UNIT=TAPE62, // DCB=BLKSIZE=3200 VI-37 ------- //DATABASE DD DSN=DBFILE1,DISP=OLD //CODEFILE DD DSN=DBFILE2,DISP=OLD //INDEX DD DSN=DBFILE3,DI5P=OLD //SYSLST DD SYSOUT=A //SYSIPT DD * parameter cards SYS009 = DD name of Journal File IDMS.JRNLFILE = data set name of Journal File 3200 = size of database page + 44 DBFILE1 = data set name of database DBFILE2 = data set name of CODE file DBFILE3 = data set name of INDEX file SYSOUT=A = printed outpout Roll-back IDMSRBCK is a diagnostic and recovery program that performs the following tasks; Prints page images from the journal file Restores 'before1 image to the database The print and restore functions may be specified independently of one another. Program functions are: IDMSRBCK reads a PROCESS card to determine whether processing will be TOTAL or SELECT. TOTAL indicates that the effects of all run-units will be erased to a specified point in time. SELECT designates that the effects of specified run-units will be erased according to console input. IDMSRBCK reads the journal tape backwards and: Optionally restores the database in whole or in part, according to whether PROCESS=TOTAL or PROCESS=SELECT is selected Optionally prints before and after images, according to whether or not the PRINT option is specified If PROCESS=SELECT is specified, the database is restored on a run-unit by run-unit basis, depending upon specifications entered from the console. The utility always produces a parameter listing that contains the contents of the input control cards (see below, 'Report Format1). VI-38 ------- The utility always produces a parameter listing that contains the contents of the input control cards (see below, 'Report Format'). In addition, the utility always produces a job checkpoint listing, including for each run-unit encountered: Program ID Time and date of activity Run-unit statistics contained within each end-of- job checkpoint IDMSRBCK maints a 'quiesce level1 counter for each program on the journal tape. The counter mechanism works as follows; An ENDJ (end-of-job) or ABRT (abort) checkpoint in any type of UPDATE mode increments the counter by one. A BGIN (beginning-of-job) checkpoint decrements the counter by one. A counter level of zero indicates that the database is in a quiesce (non-UPDATE) state at that time. Normally, the user rolls back the database to a designated quiesce point (i.e., a point at which no update run-units were in execution). It is possible to force IDMS to roll back to a non-quiesce point (level number greater than zero), but the user should be aware that one or more programs were at that point potentially changing the database. Thus, there is no guarantee of 'logical integrity1 in the database if roll back is executed to a non-zero level checkpoint. 'Physical integrity,' is guaranteed whenever the database is rolled back to any IDMS-created journal checkpoint. This is true since all system buffers are written to the journal file and database whenever a run-unit issues a FINISH command. To restore before images from the Journal File to the database and/or print page images from the Journal File, execute the program IDMSRBCK. Figure VI-18 illustrates the operation. Use the following JCL to run IDMSRBCK: // EXEC PGM=IDMSRBCK //STEPLIB DD DSN=SYS2.IDMS.LOAD,DISP=SHR VI-39 ------- PARAMETER CARO(S) IDMSR8CK USTING DATABASE riGURL VI-1U RESTORING BEFORE IMAGES ------- //SYS009 DD DSN=IDMS.JRNLFILE,DISP= (OLD, PASS), UNIT=TAPS62, // DCB=BLKSIZE=3200 //DATABASE DD DSN=DBFILE1,DISP=OLD //CODEFILE DD DSN=DBFILE2,DISP=OLD //INDEX DD DSN=DBFILE3,DISP=OLD //SYSLST DD SYSOUT=A //SYSIPT DD * parameter cards IDMS.LOAD = data set name of IDMS load library SYS 009 = DD name of Journal File IDMS.JRNLFILE = data set name of Journal File TAPE = symbolic device name of Journal File 3200 = size of database page •*- 44 bytes SYS010 = DD name of database DATABASE = data set name of data base SYSOUT = A = printed output DBFILE1 = data set name of data base DBFILE2 = data set name of CODE fileput DBFILE3 = data set name of INDEX file VI-40 ------- 6. DATABASE MAINTENANCE (1) Error Conditions In the course of running programs against the database, conditions will arise which cause the database to be in error. The conditions will include a bug in a program, a program which aborted without recovering the database properly, or a bad transmission of data. In the data base, this could result in broken set chains, improperly stored data or non-retrievable data. (2) Diagnosis The presence of any of these errors can cause subsequently run programs to executed incorrectly or to abort with mysterious error codes. If a set chain is broken, frequently the program error will be "no current of set-name". Garbled data may lead to a "record not found" or a set which appears to be out of sort. It should also be remembered that a broken set chain will not cause all accesses to a record to fail, but only those accesses which use the set in error. Therefore, the error may exist for some time before it is discovered, making it virtually impossible to pinpoint the cause of the error. This makes it impracticable to use the journal tape to roll back the program which caused the error. The major tool for diagnosis and correction is IDMSPFIX. Data Base Page Dumps - IDMSPFIX IDMSPFIX is a program debugging tool that allows the user to print and, optionally, alter data in a database page and write the altered data back to the database. Program functions are: IDMSPFIX reads a PROCESS card to determine the input source - i.e., cards or console. The utility accepts and processes one function at a time. A function is equivalent to a group of one or more cards or lines from the console. The utility prints pages in decimal, hexadecimal, or both formats, depending on PRINT parameter specification (both Decimal/Hexadecimal). In addition, pages may be printed singly or by page number range. IDMSPFIX locates and prints CALC records, using: VI-41 ------- Area name into which the record has been randomized or low and high page limit of CALC range CALC key value Optionally, additional pages following the CALC'ed page are printed. IDMSPFIX alters data and writes it back to the database using the PATCH, VERIFY, and REPLACE parameters. The user is urged to exercise caution when taking advantage of these powerful options. General syntax requirements are listed below. For specific parameter information refer to Database Design & Definition Manual, Section 11-43. first card INPUT= CARDS ,DMCL=device-media-name CONSOLE subsequent cards or console entries (printing database pages) PRINT = BOTH DECIMAL, Area-name HEXADECIMAL (printing page from calc key) AREA = Area name CALC = Key-value [,PAGES= page-count] (end of job) END The process goes to end of job. To print pages from the database and/or make patches, execute the program IDMSPFIX. Figure VI- 19 i llustrates the operation. Use the following JCL to run IDMSPFIX: // EXEC PGM=IDMSPFIX //STEPLIB DD DSN=IDMS.LOAD,DISP=SHR VI-42 ------- PARAMETER CARDS IDMSPFIX DATABASE USTING FIGURE VI-19 ALTERING AND PRINTING PAGES ------- //DATABASE DD DSN=DBFILE1,DISP=OLD //CODEFILE DD DSN=DBFILE2,DISP=OLD //INDEX DD DSN=DBFILE3,DISP=OLD //SYSLST DD SYSOUT=A //SYSIPT DD * parameter cards SYS 2.IDMS. LOAD = data set name of IDMS load library DBFILE1 = data set name of database DBFILE2 = data set name of CODE file DBFILE3 = data set name of INDEX file SYSOUT=A = printed output (3) Error Correction Operational Precautions Error Correction using the IDMSPFIX utility should be undertaken with the greatest precautions possible. Misuse of the tool will lead to a database disaster. None of the normal safety features (journaling) will be in effect to protect against careless errors. If misuse becomes a problem, the utility should be removed from the library and kept in the DBA's desk. A log should be set up to record all changes (much like a log by recording IDMS system changes sent by CULLINANE). Database Page Fixes The program IDMSPFIX has been previously described in (2) Diagnosis, Database Page Dumps. In order to alter pages of the database replace the print parameters of the IDMSPFIX with the following: PATCH = n VERIFY = aaaa pppp REPLACE = aaaa pppp FINISHED VI-43 ------- 7. SECURITY PROCEDURES (1) On-line Data Base Protection In order to protect the data base as a whole from unauthorized usage, OPP must rely on facilities within the operating system. The first operating standard which should be employed is to put a disposition of "OLD" on all database files in all IDMS application programs and utilities. Not only will this have the affect of locking out unauthorized users most of the time but it will force all IDMS jobs to be executed serially, thus insuring data base integrity in a non-CAMP environment. Additional protection can be obtained by using OS file passwords on the data base files. The data base files are the most valuable component of the OPP Monitoring systems and, as such, should receive maximum physical security. No unauthorized person should be allowed to remove them or destroy them. Furthermore, the same degree of protection should be given to the IDMS dump tapes and journal tapes. (2) Subschema Protection For those programs and their users running under the auspices of IDMS, the subschema provides maximum security to prevent unauthorized access to data within the database. The major security consideration is to protect the subschemas themselves from unauthorized changes. Since the ultimate protection would be deletion of the subschema compiler from the IDMS load library, which would prove inconvenient for those authorized users, the DBA should simply make frequent checks on the subschemas using the IDMSRPTS utility. Subschema Usage The Subschema DDL enables the DBA to define a subset of the 'global' or schema database definition. This definitional subset provides security and privacy in three ways: (i) The Subschema DDL allows the DBA to establish a 'program view,1 consisting of only those database areas, records, and sets that are relevant to the application. The program view forms the basis for data privacy in the IDMS environment. (ii) The Subschema DDL permits the DBA to 'segment' database records; i.e., define portions of records that are to be made available to the application excluding data-items not pertinent to the application. VI-44 ------- (iii) The Subschema DDL allows the DBA to define privacy locks that restrict specified area usage modes and explicit DML functions. For instance, if an area usage mode is specified as having a privacy lock of 'NO1 (see Section 8, 'Subschema DDL Preparation1), the pertinent database area may not be READIED with that usage mode. These locks permit the DBA to govern the manner in which the database resource is to be used. For example, the DBA may indicate that the CUSTOMER- REGION cannot be opened in EXCLUSIVE UPDATE mode. This specification ensures that concurrent run-units may access the CUSTOMER-REGION, but only in RETRIEVAL usage mode. A second type of privacy lock restricts explicit DML functions against specified record and set types. For example, by defining 'PRIVACY LOCK FOR CONNECT IS NO,1 the DBA may prohibit this function from being executed against specific records and sets. However, the user should note that implied functions are not prohibited. Thus, the privacy lock for CONNECT does not preclude the use of the STORE command, with its associated implied CONNECTS. Refer to Section 8 of the IDMS Programmer's Reference Guide for a discussion of subschema privacy locks. The subschema compile phase generates control tables that reflect the program view, data segmentation, and privacy lock restrictions described here. The compiler posts this information into the Directory for subsequent use by the DMLC and DMLP processors. At execution-time, these tables will provide additional security (3) Unauthorized Programs Compilation Protection Prior to execution of the IDMSDMLC or DMLP processors, the Data Directory Utility (IDMSCLUC) stores program control information in the Directory file. These controls define which programs may invoke a given subschema, and, therefore, constitute important compile-time checks. Two IDMSCLUC action codes require that a program be registered in the Directory prior to compilation: +_ (ADD/DELETE 'any-program1 option) - The -A code specifies that a program name must be registered in the Directory for a given subschema prior to compilation. +G (ADD/DELETE 'program-name') - The +G code registers the program name in the Directory. At compile-time, the DMLC and DMLP processors check for proper program registration. If the program name is not registered for the subschema, and the subschema's 'any- program' option is turned 'off,' compilation is prohibited. VI-45 ------- The DBA should note that when the "any-program" option is turned 'on1 (the default), the DMLC and DMLP processors will execute, regardless of whether the program name has been entered via the +G command. In addition, only COBOL and PL/I programs may be registered in the Directory. Assembler and FORTRAN programs are not subject to IDMSCLUC controls, since these programs are not pre-processed by the DML compilers. Additional Authorization Checks Database procedures are special DBA-coded routines that may be associated with a particular DML function for a specified record type. These modules intercept functions between the DBMS and the user program, enabling the DBA to modify data on its way into and out of the database, perform search algorithms, gather statistics, and execute various special functions. Database procedures also have several uses in the context of data privacy and security, including: Encoding/decoding of data-items and records to ensure physical data security Compression/decompression of data to save disk space and provide controls against data theft. Validation of data-items to establish data integrity Enforcement of data privacy at the record occurrence level (4) Protection of Input Files Within the OPP Monitoring System, the intermediate input files which have been created from the survey forms and are sitting on the disk prior to being added to the data base represent the least secure aspect of the system. It is therefore recommended that these files be protected with OS file passwords. 8. ARCHIVAL DATA The DBUNLOAD program can be used to either partially or fully unload the database. The data is stored into site, master, residue, and code files. Parameters to the program are stored in a parameter file and allow the removal of all the data or only that data which pertains to a sample taken prior to the specified date. VI-46 ------- |