Data Management Series Non-Relational Data Tasks

Introduction

Ever since the second computer was introduced into the world the file management tasks of data file transfer, share, convert and compare (or data file validation) have been technically challenging. This document is an overview of the data file management aspects of an application migration between a Mainframe System and a Windows System running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express (referred to in this document as a Wintel System). This white paper provides a quick overview of the data file management process and data file structures (both file format and record content). Links to documents that provide detailed information for data file transfer, conversion and comparison are included.

The tasks performed within the file management process should be repeatable with audit or validation trails. A task should be executable on demand or as a scheduled, automated, unattended process. Requiring operator input during the execution of a task within the process introduces an exposure point for error.

The structure and composition of "Business Data" and "Business Processes" as they exist in the Information Technology environment is a major factor in determining if and/or when a data conversion should be addressed. The mindset should be to explore the EBCDIC-ASCII possibilities while maintaining Mainframe Numeric Formats (or ANSI/74 Numeric Formats).

An application dependency on traditional flat files and/or VSAM Data Sets when considered as a single entity would favor staying in EBCDIC. If the file types and record structures are poorly defined and/or poorly documented then the argument for EBCDIC is even stronger. However, the lack of good data definitions and/or documentation has created a support burden in the current mainframe environment. If the decision is to stay in EBCDIC then moving the application off the mainframe will also carry this support burden into the new environment and continue to limit intra-platform and inter-platform flexibility. If this support burden is not addressed during the application migration process it will probably follow the same life cycle in the new environment as it had on the mainframe.

The requirements for printing (or Reports) need to be a part of but not limited to the EBCDIC/ASCII discussion. However; in the mid-range environments printer are ASCII-centric and usually require a different set of print control characters. Micro Focus has a number of options when dealing with printed output. The quantity and complexity of printed output could be a major factor in making an EBCDIC/ASCII decision and will have an impact on results comparison during the testing process.

A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.

We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.

Copyright © 1987-2025
SimoTime Technologies and Services
All Rights Reserved

Data File Management Tasks

There are seven tasks within a data file migration effort. The detail within the tasks is based upon the direction of migration and which system will be performing the individual tasks. The location and sequence of task execution may be predefined, required for both systems or optional to either system.

Task 1, Prepare the Two Systems

This should be a one-time process. However, consideration should be given for replicating the environment across development, test, production and backup systems.

The mainframe is the current production system. Mainframe resources are committed to production jobs and disk capacity may be a problem when attempting to copy and convert very large files. Therefore, most of the conversion jobs will be done on the Wintel platform where resources are not in contention with the production system.

1.1, Prepare the Windows Directory Structure

This may appear to be an insignificant task. However, the number and variety of files will grow during the download and conversion process because of the file transfer methodology (FTP or MFA) and the file format (sequential, Indexed, variable and fixed formats) and the file content requirements (ASCII or EBCDIC encoding schemes).

The following is a sample directory structure for a development environment.

1.2, Install and Configure Software

The prerequisite software required is as follows:

1.3, Compute Counts & Currency Totals

Record counts are a minimum requirement to ensure that all the records sent are received. Currency totals are an option and provide an extra validation. When converting between EBCDIC and ASCII encoding formats the fields within records within files that contain currency values should be tracked with a batch total for comparison unless the parallel processing done during the testing cycle will produce reports that include this information.

1.4, The Mainframe System

In most situations record counts are not readily available on the mainframe. It is not unusual to create a program that reads through the file and counts the records. If it is necessary to do this then the same program should be used to accumulate currency totals.

Another item to consider is the use of third party software that provide alternative access methods that compress data and make record counts difficult to calculate. For example, Innovation Access Method (referred to as IAM and provided by Innovation Data Processing) is a high-performance, indexed access method for OS/390 and MVS/ESA operating systems. Existing as non-VSAM data sets provides IAM with capabilities to eliminate the 4.3-gigabyte file-size restriction in VSAM prior to DFSMS V1.3 and to choose a block size that optimizes space utilization on DASD.

1.5, A Windows & Micro Focus Environment

If the record counts and currency totals are calculated during the data file conversion that is being done on the receiving system then it may be considered an optional or redundant task to count records or accumulate currency totals during the transfer process.

If the Micro Focus data file converter (DFCONV) or the SimoTime technologies are used for the file conversion then record counts are provided by both technologies. A definite advantage for accumulating currency totals is to use the SimoTime technologies since this approach does not do the conversion but creates COBOL source code that may be compiled and executed to do the conversion. These COBOL programs may be easily modified to accumulate currency totals during the file conversion process.

1.6, Documentation & Reference Material

It is very important to document the process for preparing the sending and receiving systems so it may be repeated. This is especially true for any exception type of process.

    Explore a description of the Challenges of Application Migration and Data File Management that includes data file sharing, transferring, conversion and validation or comparison.

    Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.

Task 2, Transfer the Data Files

Determine the data file transfer methodology. Mainframe Access (MFA) is recommended especially for Key Sequenced Data Sets (VSAM, KSDS) or sequential files with variable length records.

2.1, File Transfer Protocol (FTP)

In many situations this is the protocol by default since it may be the only thing that is common to the Sending and Receiving systems.

If the FTP transfer technology is used then all non-sequential files will need to be converted to sequential files. Special handling will also be needed for sequential files that contain variable length records in order to retain the record descriptor word (RDW) that precedes each record and contains the record length.

2.2, Mainframe Access (MFA)

This technology is available from Micro Focus and in most cases is the easiest way to transfer files from a mainframe to a Windows system running a Micro Focus sub system. For data files (i.e. non-source code) the received files will arrive in a Micro Focus format and the file content will be EBCDIC encoded. If Micro Focus Mainframe Access (MFA) technology is used then very little preparation will be required to transfer the data files.

2.3, Machine Readable Media

The decision to use this media is determined by file sizes and availability of hardware that is accessible from both the sending and receiving system.

2.3.1, Tape, Compact Disk (CD) or Shared Disk

This conversion may only be a file format conversion that addresses the media exchange issue and may require additional file format conversion and file content conversion once it is transferred to the Wintel system.

2.3.2, Sub-Contract or Out Source

This is usually providing a sub-contractor or data conversion company with copies of the data files on tape in an agreed to format. The files are converted to a compatible format and delivered on a media that is accessible by the hardware available on the Wintel system. This conversion may only be a file format conversion that addresses the media exchange issue and may require additional file format conversion and file content conversion once it is transferred to the Wintel system.

2.4, Create Scripts for File Transfer

On the mainframe batch jobs (JCL) will need to be created to prepare the files for transfer. Programs may need to be created to accumulate currency totals. On the Wintel system this is usually a suite of command files (previously known as batch files) that will be called from a mainline command file. Creating the scripts should be a one-time process.

2.5, Execute Scripts, do File Transfers

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. Depending on the file transfer medium, the file types and the validation requirements the mainframe files may need to be pre-processed (or staged) along with the control information such as record counts and hash totals for critical currency fields. Once the files are staged then start the transfer process between the Mainframe system and the Wintel system.

For additional information refer to the Data File Transfer document provided with this series of documents.

Task 3, Convert the Data Files

Since Net Express includes a Data File Converter (DFCONV) this may be used at no additional cost. However, this limits the data file conversion process to being executed on the Windows platform. Also, if the record structures of a file are multiple record types with complex structures then an alternate solution such as SimoTime technologies may be required.

3.1, Prepare Specifications & Control Media

Create specification media for Data File Conversion Utility (DFCONV) provided by Micro Focus. This is a one-time task. If required, create specification media scripts (or control media) for SimoTime Technology to generate, compile and execute the data file conversion programs. This should be a one-time task. The scripts that set the environment and perform the execution of the conversion programs should be written to be an unattended and repeatable process.

3.2, Create Scripts for Convert Process

This is a one-time task. The number of scripts required depends on the number of files, the size of the files and the complexity of the record structures within the files. For the Wintel system a script is usually a command file that will convert a group of files or a single file. A mega script may then be created to call the individual scripts.

3.3, Execute the Data File Convert Process

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. The SimoTime technology does not explicitly provide the capability of accumulating currency totals. However, since the SimoTime technology generates COBOL source code that is compiled and then used to do the data file comparison this COBOL source code may be easily modified to accumulate the batch totals. For additional information refer to the Data File Convert document provided with this series of documents.

Task 4, Validate Counts & Totals

Validating record counts between the sending and receiving system is part of the basic requirements to ensure the files being transferred and converted will have an equal number of records.

Actual data file compares at the record level are not done at this point in the process. Only validation of record counts and possible subtotals of currency fields will be done at this point in the process. The goal at this point in the process is to validate that the number of records sent by the mainframe is equal to the number of records received and converted at the Wintel system.

The method for obtaining record counts on the mainframe or the Wintel platform varies widely depending on the software used to do the transfer or the conversion. The SimoTime convert technology provides record counts as part of the conversion process.

This suite of programs and documentation will describe and demonstrate an approach for reading a file, calculating record counts and producing summary totals for a set of pre-defined numeric fields. The approach uses two COBOL programs that were generated using SimoTime technology. The programs are generated on a Windows System with Micro Focus COBOL and may be compiled and executed on an IBM Mainframe or a Linux, UNIX or Windows System with Micro Focus COBOL.

Task 5, Process Parallel Tasks

This requires taking a backup or transferring the data files of the mainframe or sending system prior to starting the processing (or execution of parallel tasks or jobs). Once the pre-process files are transferred the jobs that process the data may be executed on both systems. When the processing is completed on the mainframe the post-process files will then be transferred to the Wintel platform and then converted and compared with the post-process files on the Wintel system.

5.1, Execute Test on the Mainframe System

This is a three-step process.

5.2, Execute Test on the Windows System

This is a two-step process.

Proceed with the task of comparing the files.

Task 6, Compare the Results

One of the early decisions is to determine the platform where the data files will be compared. Some of the questions or considerations are as follows

For example, it may be necessary to sort some files (this is true if the files are in sequence by an alpha-numeric field). In addition to comparing the files at the record level there should be batch record count comparisons and possibly some subtotal comparison on critical currency fields.

The SimoTime compare technology has an option to read to "End-of-File" or to "Quit" after a certain number of differences have been identified.

If the "End-of-File" option is used then differences will be written to the log file and when the difference count exceeds the maximum then differences will not be written to the log file but reading of the two files being compared will continue until end-of-file is reached and a records count for the total number of records read from each file will be provided along with a non-zero return code.

If the "Quit" option is used and the maximum number of differences is exceeded then a count of the number of records read before the program is terminated with a non-zero return code will be provided.

It may be a requirement to accumulate batch totals for currency fields. The SimoTime technology does not explicitly provide this capability. However, since the SimoTime technology generates COBOL source code that is compiled and then used to do the data file comparison this COBOL source code may be easily modified to accumulate the batch totals.

6.1, Prepare Specifications & Control Media

Create specifications media and scripts (or control media) for the SimoTime Technology to generate, compile and execute the data comparison programs. This should be a one-time task. The scripts that set the environment and perform the execution of the compare programs should be written to be an unattended and repeatable process.

6.2, Create Scripts for Compare Process

This is a one-time task. The number of scripts required depends on the number of files and the positions within the records that will be compared. For the Wintel system a script is usually a command file that will compare a group of file sets or a single file set (a file set consist of two files). A mega script may then be created to call the individual scripts.

6.3, Execute the Compare Process

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. Review the log files for possible differences For additional information refer to the Data File Compare document provided with this series of documents.

Task 7, Review Results

Based on the results the next steps would be one of the two following choices.

7.1, Proceed Forward

If the compare process indicates the files from the mainframe process match with the files created by the Wintel process then it is time to move to the next level of testing or begin the process of completing the final task of moving to a production system on the Wintel system.

7.2, Correct & Repeat

If differences in the results are encountered then it will be necessary to remove the cause of the differences and repeat the testing prior to moving forward.

Summary

Migrating an application between systems of similar architecture (i.e. upgrade to existing mainframe) can be a challenging task. Migrating an application from a Mainframe, EBCDIC encoded system to a Wintel, ASCII encoded system requires an even higher level of planning and attention to detail.

With careful planning and a documented process the application migration process can move forward to a successful conclusion. This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.

In the world of programming there are many ways to solve a problem. This documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please contact us using the information in the  Contact or Feedback  section of this document.

Software Agreement and Disclaimer

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.

SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Downloads and Links

This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.

Note: A SimoTime License is required for the items to be made available on a local system or server.

Current Server or Internet Access

The following links may be to the current server or to the Internet.

Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.

    Explore a Series of White Papers for non-relational data files. This includes information about data file management in a diverse or mixed systems environment.

    Explore a quick overview of the data file management tasks for data file transfer, conversion and comparison. Ever since the second computer was introduced into the world the file management tasks of data file transfer, share, convert and compare (or data file validation) have been technically challenging.

    Explore the alternatives for transferring data files between systems. This link provides access to a repository of information that includes the transferring and/or sharing of data between Mainframe (ZOS or VSE), Linux, UNIX and Windows Systems.

    Explore the Principles of Data File Conversion. This link includes guidelines for defining requirements and determining the scope of effort for a data conversion effort.

    Explore the Principles of Data File Comparison. This link includes guidelines for defining requirements and determining the scope of effort for a data comparison effort.

    Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.

    Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.

    Explore The File Status Return Codes that are used to interpret the results of accessing VSAM data sets and/or QSAM files.

    Explore The Binary or COMP format for numeric data strings. This numeric structure is supported by COBOL and may be explicitly defined with the "USAGE IS COMP" or "USAGE IS BINARY" clause.

    Explore The Edited for Display format for numeric data strings. This numeric structure is supported by COBOL and may be used with an edit-mask to prepare the presentation for readability by human beings.

    Explore The Packed-Decimal or COMP-3 format for numeric data strings. This numeric structure is supported by COBOL and may be explicitly defined with the "USAGE IS COMP-3" clause.

    Explore The Zoned-Decimal format for numeric data strings. This numeric structure is the default numeric for COBOL and may be explicitly defined with the "USAGE IS DISPLAY" clause.

    Explore commonly used formats and processing techniques for managing various numeric formats available on the mainframe.

    Explore the Numbers Connection for additional information about the structure and processing of numeric data items (or numeric fields).

Internet Access Required

The following links will require an internet connect.

A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection

Explore The Micro Focus Web Site for more information about products (including Micro Focus COBOL) and services available from Micro Focus. This link requires an Internet Connection.

Glossary of Terms

    Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.

Contact or Feedback

This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.

We appreciate hearing from you.

Company Overview

SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.

Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.

Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com