Applications & Data
Migration, Management & Data Integrity
  Table of Contents  v-16.01.01 - aplmig01.htm 
  Introduction
  Application & Data Migration Process
  Application & Data Characteristics
  Consulting & Training Options
  System, Sub-System & Utilities
  Security
  Workload Automation, Batch Jobs
  Report Management and Printing
  Application Asset Migration
  Prepare a Windows Environment
  Transfer Source Members
  Micro Focus Mainframe Access (MFA)
  File Transfer Protocol (FTP)
  Machine Readable Media
  Compile Source Members
  Data File Migration
  Prepare a Windows Environment
  Directory Structure for Windows
  Data File Transfer
  FTP or File Transfer Protocol
  MFA or Mainframe Access
  MRM or Machine Readable Media
  Data Migration, File Conversion
  File Format Conversion
  Sequential with Fixed Length Records
  Sequential with Variable Length Records
  KSDS to Micro Focus Key Sequenced
  Record Content Conversion
  ASCII and EBCDIC
  Maintain Numeric Integrity
  Data File Comparison
  Data File View & Edit
  Data File Formats (Micro Focus)
  Data File Migration Process
  Archived or Historical Data
  AS/400 as a Client
  Automated & Repeatable Test Cycle
  Anticipate Compare during Convert
  Validate & Audit after Completion
  Summary
  Software Agreement and Disclaimer
  Additional Reference Materials
  Numeric Field Formats
  Data File Conversion
  Data File Comparison
  Data File Transfer
  Non-print or Non-display Characters
  Vendors, Consultants & Service Providers
  Glossary of Terms
  Comments or Feedback
  Company Overview
The SimoTime Home Page 

Table of Contents Previous Section Next Section Introduction

Ever since the second computer was introduced into the world the tasks involved with a System Upgrade or replacement, an Application Migration and Data File Management (data file sharing, transfer, conversion and validation or compare) have been technically challenging.

The term "Application Migration" may be defined as the moving of all the components of an application from an "existing system" to a "new system" and obtaining the same business results. The term "system" is used to refer to existing or new hardware, operating system software and sub-systems such as online transaction monitors or job schedulers. The system provides the foundation for the application to execute and deliver the critical business requirements.

If changes become necessary then identifying the "Application Change" tasks and making the changes after a successful migration would help differentiate "risk incurred through application change" vs. "risk incurred through application migration or system dependencies". Also, this process will identify and help to manage the "scope creep" that may occur during an "Application Migration" effort.

Maintaining Data Integrity during an application migration and data file conversion process should be the top priority. To succeed requires careful planning and analysis with constant monitoring of the data transfer, data sharing, data conversion and data validation processes.

This document discusses the "possibilities and considerations" aspects of an application migration and will focus on providing an overview of the data management process during a mainframe application migration effort.


We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.

Copyright © 1987-2017
SimoTime Technologies and Services
All Rights Reserved

Table of Contents Previous Section Next Section Application & Data Migration Process

The following is a list of the steps for an application migration effort.

Item Description
1 Planning and Analysis .
2 Design, Build and Configure the Receiving System
2.a  Hardware
2.b  System Software
2.c  Sub-Systems and Supporting Software
3 Migrate the Application
3.a  Transfer and Compile the Code
3.b  Transfer and Convert the Data
3.c  Unit Test the Application
4 Application and System Testing
4.a  Proper Program Execution
4.b  Data Integrity
4.c Support Structure (Helpdesk)
5 Results Validation (Data File Compare)
6 Deployment (Monitor and Support)
  Application and Data Preparation and Migration, a List of Tasks

When moving from the "application migration and unit testing" phase to the "application and system testing" phase the support requirements will change from a planned, proactive-oriented process to an on-demand, reactive-oriented process. During the migration-oriented phase we had hours or days to plan and resolve issues. As an alternative we could possibly bypass the issue (develop a temporary work around) until a fix was provided.

During the Testing phase individuals doing the testing will be reporting problems as they occur and will be expecting a quick response so they may continue with their testing effort. When a problem is identified in the testing phase it must be resolved quickly with a permanent, long-term fix (a work around or temporary fix with a follow-on permanent fix may invalidate the testing). If an issue is encountered that stops the testing from moving forward it must be resolved within a matter of minutes or hours. If the issue is not resolved in this timeframe it could result in a rescheduling of the testing phase (this could possible require a partial or complete synchronization of the data and other system resources).

Therefore, it is important to have the training for the support personnel completed and involved in the support process prior to starting the "application and system testing". This will provide the primary support required and offer an opportunity to gain hands-on experience with the application running in the new environment.

Table of Contents Previous Section Next Section Application & Data Characteristics

Many companies have successfully migrated business applications between Mainframe Systems and a Windows, UNIX or Linux Systems. Based on these successes companies are now motivated and actually pursuing the migration of additional applications and data between a Mainframe System and a Windows, UNIX or Linux System.

These additional applications continue to increase in size and complexity along with the transfer, share, convert and compare of data that is formatted in an ever-increasing variety of file structures, formats, encoding schemas, data bases and access methodologies.

The Mainframe System continues to evolve and customers are migrating applications that are an intermingling of reliable, twenty-year old programs and file structures combined with programs and data structures that incorporate the latest mainframe technologies.

Mainframe applications that are created using the COBOL programming language are referred to as "COBOL-oriented applications" in this document. Batch jobs require the use of JCL (Job Control Language) and on-line programs usually interface with a transaction system (for example CICS, Customer Information Control System) and screen handlers such as BMS (Basic Mapping Support).

Therefore, a mainframe application may be predominately COBOL but will have other source members such as JCL or BMS (hence the term COBOL-oriented-application vs. COBOL-application).

In addition, many years ago (prior to the release of the ANSI/85 standard and the release of IBM's COBOL/2 for the mainframe) the COBOL language did not have the functionality to meet some of the programming requirements needed to meet the business and system demands. The COBOL Compilers did not generate executable members that were finite-in-purpose and fast enough to meet the size and performance requirements. To solve these problems Mainframe Assembler programs and callable assembler routines were written and many are still in use today. As the mainframe has evolved the capability of Mainframe Assembler has evolved. Mainframe Assembler continues to be used to address "Systems-oriented" requirements resulting in the combined use of older and newer Mainframe Assembler technologies.

The key to a successful application migration (and obtaining platform flexibility) is the ability to recreate the system and sub-system foundations that will support the execution of the COBOL programs along with the non-COBOL programs and manage the data using the existing application methodologies for a batch or online environment.

Table of Contents Previous Section Next Section Consulting & Training Options

The following is an overview of some of the services and training options available for an application migration effort. The following is a list of the recommended consulting, services and training phases designed to minimize the risk and successfully complete an application and data migration effort.

Item Description
1 The Analysis & Planning Phase
2 Initial Training & Product Installation
3 The Data File Management & Migration Phase
4 The Transfer, Development & Support Phase
5 Testing & Quality Assurance Phase
6 Production System Configuration & Deployment Phase
7 Review, Improve and Adjust
  Overview of the Phases of an Application and Data Migration Project

Training will be addressed in the Analysis and Planning phase based on individual assignments and requirements. Each of the options is discussed in more detail in the following sections of this document.

When moving from the "application migration and unit testing" phase to the "application and system testing" phase the support requirements will change from a planned, proactive-oriented process to an on-demand, reactive-oriented process.

During the migration-oriented phase we had hours or days to plan and resolve issues. As an alternative we could possibly bypass the issue (develop a temporary work around) until a fix was provided. During the Testing phase individuals doing the testing will be reporting problems as they occur and will be expecting a quick response so they may continue with their testing effort. When a problem is identified in the testing phase it must be resolved quickly with a permanent, long-term fix (a work around or temporary fix with a follow-on permanent fix may invalidate the testing). If an issue is encountered that stops the testing from moving forward it must be resolved within a matter of minutes or hours. If the issue is not resolved in this timeframe it could result in a rescheduling of the testing phase (this could possible require a partial or complete synchronization of the data and other system resources).

Table of Contents Previous Section Next Section System, Sub-System & Utilities

Migrating an application from a mainframe requires lifting the source code for the application off of its current foundation of hardware (mainframe), system software (z/OS or VSE), sub-system (CICS) software and Data Base Management (DB/2) or File Management (VSAM or sequential files) software.

The source code for the application is then shifted to a new foundation of hardware (intel), system software (Windows), sub-system (Micro Focus ES/MTO with Net Express for development) and Data Base Management (MS/SQL) and File Management (Micro Focus for Indexed and Sequential files) software. The source code for the application is then compiled and we are almost ready for our first unit test.

At this point it is important to note that Micro Focus provides the following.

Item Description
1 Many of the utilities (such as SORT, IEBGENER, IDCAMS, IEFBR14 and more) that are used on the mainframe are provided on the Linux, UNIX and Windows platforms.
2 A subset of routines for the LE (Language Extensions) callable functions.
3 The capability of running an application on a Linux, UNIX or Windows platform in an EBCDIC or ASCII encoded environment.
  Micro Focus Capabilities or Alternatives

Refer to the Micro Focus documentation for more detail.

Third-Party software may be application oriented, sub-system oriented or utilitarian. Many software vendors now offer a Windows or Distributed version of the mainframe software packages. For example, a third-party package that manages and catalogs reports into a database and provides easy access by a business user would be sub-system oriented. A third-party package that provides a file copy function and is called from a JCL submission or command file execution would be utilitarian.

Table of Contents Previous Section Next Section Security

For an "Application and Data Migration" project security requires a two-phased approach. The Micro Focus Enterprise Server, Mainframe Transaction Option (ES/MTO) provides sub-system level security in the Windows environment. To obtain system level security it will be necessary to use a combination of ES/MTO subsystem security and Active Directory Security provided by Microsoft. This is a separate discussion that is beyond the scope of this document.

Table of Contents Previous Section Next Section Workload Automation, Batch Jobs

A number of software vendors including some of the following vendors now offer distributed (or Enterprise) products for Windows, UNIX and Linux (distributed) platforms. If the job scheduling requirements are relatively simple the Task Scheduler provided with Windows may satisfy the scheduling requirement.

ASG-Zena is a robust, enterprise-wide workload management solution for distributed operations environments that support “event-based” scheduling as well as traditional date- and time-based scheduling methodologies.

BMC Control-M simplifies IT job scheduling across the board. While its features are advanced – automated conversion wizard, customizable self service, application integration – running it couldn't be easier. BMC Control-M is simple to install, simple to get going, and simple for your business to use.

CA Workload Automation enables a user to schedule and manage workload processing across your entire IT environment. Using CA Workload Automation you can update workload processing schedules in real-time to accommodate changing business requirements.

Cisco Tidal Enterprise Scheduler simplifies job scheduling, and automates and tracks how jobs are carried out. Tidal Enterprise Scheduler provides a single point of control and visibility for job scheduling, managing, and performing all scheduling processes.

IBM® Tivoli® Workload Automation enables you to easily manage composite business and IT workloads that have complex dependencies spanning multiple applications, systems including mainframe, distributed and high-performance grid computing environments.

ORSYP technology monitors production through a single management console – and even offers a web interface for remote access. A single enterprise-class automation solution reduces the risk and costs associated with managing and maintaining multiple products and tools while enabling proactive management on critical issues. Automating IT Operations with ORSYP increases productivity, service quality and more efficient usage of existing resources.

UC4 Software delivers a unified automation solution that intelligently orchestrates your business processes, applications, and IT infrastructure. UC4 a single enterprise-wide approach that intelligently integrates the automation of business processes, applications and IT infrastructure onto one platform.

The use of a Job Scheduler will require additional effort when Migrating an application to a distributed platform. This effort usually requires a replacement product (i.e. a vendor's distributed scheduler instead of the vendor's Mainframe scheduler) that will need to be installed and configured for the targeted distributed platform. It is important to note that many job schedulers provide additional capability beyond the simple scheduling of jobs. Jobs may be scheduled as follows.

Item Description
1 On demand
2 To run at a specified time
3 Based on an event
4 Based on a condition
5 Based on the presence of a file or other entity
6 and more …
  Job Scheduling Requirements or Dependencies

A job scheduler may also do parameter substitution. For example, the use of the %argument syntax may be used as a substitution capability where the substitution value for the %argument is stored in an external table. This capability functions similar to the capability of JES to define and substitute values based on the &argument syntax to identify where the substitution is to take place. The difference is the value for the &argument is specified in the JCL stream and the value for %argument may be specified in an external table or database.

A job scheduler should be able to monitor job progress and take appropriate action based on certain conditions. A job scheduler can also track system usage based on user (or other criteria) that may be necessary for billing.

Table of Contents Report Management and Printing

This section will discuss Report Management Systems and Printing. Third Party Vendors supply sub-system or utilitarian software to extract printed reports from the mainframe output spool and place the reports into a repository for easy access by business users. Mobius Management Systems is one of the vendors that supply this type of software in both a mainframe and a distributed (i.e. Windows) environment.

VPSX® software (provided by Levi, Ray & Shoup, Inc.) is a robust, scalable Enterprise Output Management System for UNIX, Linux, Windows, and other distributed environments. Designed to capture data from any platform, VPSX spools that data, then delivers the output to printers, fax servers, file servers, and other devices.
VPSX was designed from the ground up for flexibility and ease of use. When required, VPSX can use external filters to convert data streams from a given source format to the data stream required by the target device or application. Administrators and users can monitor print status, identify and resolve errors, and control the entire printing environment using a flexible Web browser interface.

A data string that is to be printed should not contain binary data (i.e. non-print characters). The binary characters may be the same value as a print control character that could result in the incorrect presentation of the data. For additional information about non-print character processing refer to the Additional Reference Materials section of this document.

Table of Contents Application Asset Migration

This section provides an overview of the tasks that will need careful planning in order to ensure a successful application asset migration (i.e. transfer and compilation) of the source code used to build the application. Most mainframe environments have implemented a Software Configuration Management (or SCM) system.

Depending on the method used to transfer the source code between the Mainframe system and the Windows system it may be necessary to "check-out" a copy of the source members in the SCM to members in a Partitioned Data Set (or PDS). Once the source members are stored in a PDS they may be accessed as EBCDIC-encoded, record sequential files with 80 byte records. The files may be transferred between the mainframe and the Windows system using the File Transfer Protocol (FTP).

Source code members should not contain embedded hexadecimal characters. Refer to Hexadecimal Characters in Source Code for more information about managing embedded hexadecimal characters in source code.

Table of Contents Prepare a Windows Environment

A variety of directory structures or configurations are available in the Windows environment. This section will describe a configuration that works well in the Windows environment and is complimentary to the Micro Focus environment and supports the concurrent existence of a development, test and production environment on a single Windows system or spread across multiple systems. The following is the sub-directory structure used by SimoTime for the sample programs. The primary directory is C:\SimoSAM1.

               
SIMOSAM1
See Note-1
   
   
 
 
DEVL
See Note-2
   
   
 
 
ADM1
 
 
and MORE
Administrative and Support functions
   
   
 
 
ASM
 
 
and MORE
Mainframe Assembler Source and Sub-Directories
   
   
 
 
BMS
Contains Basic Mapping Support Member
   
   
 
 
COBCPY1
Contains COBOL Copy Members
   
   
 
 
COBCPY6
Contains COBOL Copy Members from a BMS GEN
   
   
 
 
COBOL
Contains COBOL Source Members
   
   
 
 
COBOLUT1
Contains COBOL Source Members, see Note-3
   
   
 
 
COBOLUT2
Contains COBOL Source Members, see Note-4
   
   
 
 
COBOLUT3
Contains COBOL Source Members, see Note-5
   
   
 
 
DATA
Contains the catalog and data files
   
   
   
   
 
 
APPL
Contains non-Relational data files for Application
   
   
   
   
 
 
SPOOL
Contains the JES and SYSOUT files
   
   
   
   
 
 
and MORE
Link to additional sub-Directories
   
   
 
 
DIRS
Contains compiler directives files
   
   
 
 
DOCS
Contains user or application documentation
   
   
 
 
HOLD
 
 
and MORE
Contains new load members from Scripted Builds
   
   
 
 
IMSLIB
Contains the IMS Members for execution-time
   
   
   
   
 
 
DBD
Source Members for the IMS DBD
   
   
   
   
 
 
PSB
Source Members for the IMS PSB
   
   
 
 
JAVA
Contains JAVA Source Members
   
   
 
 
JCL
Contains the JCL Source Members
   
   
 
 
JCLINC
Contains the Include Members for JCL
   
   
 
 
LOADLIB
Contains the executable members
   
   
   
   
 
 
simpacks
Contains Java Class Members
   
   
 
 
LOGS
Contains the log files for the application
   
   
 
 
PARMLIB
Contains the parameter or control files
   
   
 
 
PROCLIB
Contains the Procedures (or PROC's) for JCL
   
   
 
 
REXXLIB
Contains the REXX Program Members
   
   
 
 
SIMOGENS
 
 
and MORE
Contains the Program Generation Members
   
   
 
 
SYS1
Systems directory, not exposed to users
   
   
 
 
CONFIG
System configuration specifications
   
   
 
 
and MORE
Link to additional sub-Directories…
 
Note-1: This is the primary directory for the SimoTime Driver Programs and Test Cases
Note-2: This is the Development, Production or Test directory, for this example it is set to DEVL.
Note-3: Utility programs using a mainframe dialect, not part of application business functions
Note-4: Utility programs using a Micro Focus dialect, not part of application business functions
Note-5: Utility programs with specialized or unique requirements
The Directory Structure used in the in the SimoTime Development Environment

Additional detail about the structure for the ADM1, DATA, HOLD or SYS1 directories may be accessed by clicking on the "and more" item.

Color Associations: The  light-green  boxes are unique to SIMOTIME Technologies using the Mainframe and Micro Focus Enterprise Server/Studio (or Micro Focus Mainframe Express). The  light-red  boxes are unique to the SIMOTIME Technologies using a Linux, UNIX or Windows System and COBOL Technologies such as Micro Focus. The  light-yellow  boxes are SIMOTIME Technologies, Third-party Technologies, decision points or program transitions in the processing logic or program generations. The  light-blue  boxes identify the input/output data structures such as Documents, Spreadsheets, Data Files, VSAM Data Sets or Relational Tables. The  light-gray  boxes identify a system function or information item.

Table of Contents Transfer Source Members

The section briefly describes three of the popular methods for transferring source code between a Mainframe System and a Windows, Linux or UNIX System.

A separate Data File Transfer Document is available on the Internet and provides additional information about Data File Transfer process and the alternatives that are available.

Table of Contents Micro Focus Mainframe Access (MFA)

When transferring source code from a Mainframe system to a Windows system running a Micro Focus sub-system this should be the preferred transfer method. Since this is a Micro Focus utility the file format conversion (Mainframe file format to Micro Focus file format) is done as part of the file transfer process. The transfer process may be performed on demand using the Graphical User Interface (GUI) with a point-and-click or drag-and-drop methodology. The transfer process may be scripted and automated using a Windows command file, a job scheduler and the MFA command line interface.

Table of Contents File Transfer Protocol (FTP)

This file transfer methodology is usually available on Mainframe systems (both z/OS and VSE), Windows systems, UNIX Systems and Linux systems. Because it is readily available it is usually the default transfer technology.

FTP works quite well when transferring source members that are stored in a Partitioned Data Set (PDS) as sequential files with fixed-length records of eighty characters of printable text. FTP in ASCII mode will do the file transfer, the record content (i.e. EBCDIC and ASCII) and the file format (record sequential to line sequential) conversions.

Table of Contents Machine Readable Media

Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fiber Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.

Table of Contents Compile Source Members

For the program development environment Micro Focus uses the Microsoft Visual Studio or Eclipse integrated development environments (IDE's) to interactively edit, compile and debug application programs. Information about these products are available from the suppliers.

A scripted build process may be a requirement in larger environments. A predefined and documented process for doing the build of a production system without operator intervention is a minimum requirement for the audit or validation process.

This document will display information to the console and write information to a log file based on the results of the execution of the scripted compile and/or generate processes. The processes will focus on building a test or production system for the ASCII-encoded environment.

Table of Contents Data File Migration

This section provides an overview of the tasks that will need careful planning in order to ensure a successful data migration (i.e. transfer and conversion) of the data files used by the application.

Table of Contents Prepare a Windows Environment

The three tasks that are necessary to prepare are listed below and discussed in the following sub-sections of this document.

Item Description
1 Transfer the Data Files to the New System
2 Convert the Files (both File Format and Record Content)
3 Create Catalog entries to be used with Micro Focus Server.
  Preparing the Environment and Data Files

Table of Contents Directory Structure for Windows

The following shows the sub-directory structure for the DATA directories.

Note:  The Catalog used by the Micro Focus Enterprise Server is located in the DATA directory.

             
BASECAT sub-DIR's file.EXT Comments
DATA
Base Directory for non-Relational Data
   
   
   
   
APPL
 
 
.DAT
Application, non-Relational Data Files
   
   
   
ASC1
Holding directory for ASCII encoded files
   
   
   
EBC1
Holding directory for EBCDIC encoded files
   
   
   
FTP1
Holding directory, data received via FTP
   
   
   
SPOOL
 
 
MFE*.DAT
Contains SPOOL and SYSOUT info, see Note-1
   
   
   
STRS
   
   
.STR
Contains STR files, see Note-2
   
   
   
.HTM
HTML Documents for Record Structures
   
   
   
TXT1
 
 
.TXT
Holding directory for ASCII/Text files
   
   
   
WRK1
 
 
.***
Holding directory for Work or Temp files
   
   
   
XLSS
   
   
.XLSS
Holding directory for Excel, XLSS files
   
   
.CSV
Holding directory for Excel, CSV files
 
Note-1: This directory is identified by the ES_ALLOC_OVERRIDE Environment Variable.
Note-2: This directory contains the Micro Focus generated STR (or Structure) Files and the SimoTime generated HTML documents for Record Structures based on a COBOL copy file.
Directory Structure for non-Relational Data Files

Refer to the Directory Structure for a Development Environment for more information about a complete directory structure

Table of Contents Data File Transfer

There are a number of possibilities but this section will focus on three common methods for transferring data file.

Item Description
1 File Transfer Protocol or FTP
2 Micro Focus Mainframe Access or MFA
3 Machine Readable Materials
  Alternatives for Data File Transfers

A separate Data File Transfer Document is available on the Internet and provides additional information about Data File Transfer process and the alternatives that are available.

Table of Contents FTP or File Transfer Protocol

This is the transfer methodology used by default because of its availability on most systems. It can easily transfer ASCII/Text files and has the capability to transfer record sequential files in binary mode.

To transfer VSAM, key-sequenced-data-sets (KSDS) requires the file to be converted to a sequential file (using IDCAMS and REPRO). The sequential file is then transferred to the target system. The received file is then used as the base for an EBCDIC to ASCII conversion with the output file being created as a Micro Focus Keyed-Index File.

Table of Contents MFA or Mainframe Access

Mainframe Access (or MFA) provides the simplest method for transferring files from a z/OS mainframe to a Wintel, Micro Focus environment. The "File Format" conversion is handled by the MFA transfer process. The file transfer may be a simple "Drag-and-Drop" task from the GUI interface or an automated, batch process executed from a Windows command line.

Table of Contents MRM or Machine Readable Media

Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fiber Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.

For ongoing data sharing and transferring the use of removable media (tape or Compact Disk) that is available and compliant with both the Mainframe system and the Windows system is declining. The use of tape remains the popular media for backup on the mainframe.

The MRM approach usually requires file format conversions to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.

Table of Contents Data Migration, File Conversion

Refer to Additional Reference, Data File Conversion section of this document for additional information about File Format and Record Content conversion.

Table of Contents File Format Conversion

During an "Application and Data Migration" effort changing file formats should only be done when required by the receiving system or by the file transfer process. The following sections will provide a quick overview of some of the items that need to be considered when doing file format conversions.

Table of Contents Sequential with Fixed Length Records

These files are also referred to as "Record Sequential" files and may be downloaded (via FTP) in binary format and then converted from EBCDIC to ASCII encoding at the field within record level.

If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file. The received file is then used as the base to convert from EBCDIC to ASCII.

Table of Contents Sequential with Variable Length Records

The format of a sequential file with variable length records that resides on the mainframe is very different from a sequential file with variable length records that resides on a Linux, UNIX or Windows System with Micro Focus Enterprise Server.

On the mainframe the length of the individual records within a file is maintained in a Record Descriptor Word (or RDW). Special syntax is required by the FTP process to get the RDW information that contains the length of the individual records within the file. Once the file is received it can only be processed using byte-stream I/O.

If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file. The received file is then used as the base to convert from EBCDIC to ASCII.

Table of Contents KSDS to Micro Focus Key Sequenced

If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file followed by an EBCDIC to ASCII conversion.

If FTP is used to transfer the file it must first be converted to a sequential file on the mainframe and then the sequential file is transferred to the Windows or UNIX platform. The received sequential file is then used as the base to convert from EBCDIC to ASCII and create a new Micro Focus Indexed file. It is important to note that if the key field is alpha-numeric it will be necessary to provide for the difference in the EBCDIC and ASCII collating sequence.

Table of Contents Record Content Conversion

During an "Application and Data Migration" effort the typical conversion is to change the content of each record in a file from an EBCDIC-encoded format to an ASCII-encoded format. Because the record structures usually contain fields that are not just text strings (i.e. numeric packed-decimal, signed numeric zone-decimal or binary) it is necessary to do the conversion at the field level.

The Micro Focus Data File Converter (DFCONV) is used for the files with record structures that are defined by a single copy file. It is important to note that DFCONV is a developer's tool that is included with Mainframe Express and Net Express. It is not available on UNIX or with Enterprise or Application Server.

The file conversion technology that is available from SimoTime Enterprises is used for the following.

Item Description
1 Complex record structures dependent on user logic
2 Data conversion that are performed in the Linux or UNIX environments
3 Data conversion that are performed in the ZOS or VSE Mainframe environments
4 Ongoing data conversion requirements in the production environment
  List of Possible Systems where the Conversion will need to Execute

Refer to Additional Reference, Data File Conversion section of this document for additional information about file format and record content conversion.

Table of Contents ASCII and EBCDIC

Doing a conversion between ASCII and EBCDIC is a simple process. Knowing what to convert is the challenge. Understanding the record structure within a file is the key to doing a successful data file conversion at the record and field level. For applications that are currently running on a Mainfame System the definitions for record structures are usually available in the form of a COBOL copy file.

The simple guideline is to convert the alpha-numeric, text-oriented fields and do not convert the binary fields. Numeric fields will need additional considerations.

Table of Contents Maintain Numeric Integrity

The three most common numeric types (or formats) are Zoned-Decimal, Packed-Decimal and Binary. Each of the numeric types require special considerations and cannot be converted using the same techniques used for alpha-numeric, text-oriented fields.

Refer to Additional Reference, Numeric Field Formats section of this document for additional information about the various numeric types..

Table of Contents Data File Comparison

The file comparison technology that is available from SimoTime Enterprises is used to do data file compares on the Windows platform. This technology does not actually do the file compares but generates a COBOL program that performs the file compares. This COBOL program may be compiled and executed on an IBM Mainframe (z/OS or VSE), a Windows platform with Micro Focus or a Linux or UNIX platform with Micro Focus.

Refer to Additional Reference, Data File Comparison section of this document for additional information about comparing data files on a record by record basis.

Table of Contents Data File View & Edit

The Micro Focus Data File Editor is included with Mainframe Express and Net Express. The Data File Editor allows a user to view the contents of a file at the record level using an EBCDIC or ASCII translation of the data. A hex view is also provided and is very helpful if a record within the file contains packed or binary fields.

It is also possible to map record layouts to a COBOL copy file. This can be very helpful for both viewing and modifying data for testing purposes.

Table of Contents Data File Formats (Micro Focus)

This section describes various file formats and the file handling environments used by Micro Focus. For the development environment the base file handler may be the preferred environment. For the Test and Production environment the Micro Focus File Share running on a separate server may be the preferred environment for Indexed Files.

Microsoft has various formats for disk. The FAT (or File Allocation Table) is the older technology used prior to Windows 2000 and many external storage devices are shipped with the initialization being FAT. The FAT format has a limit of 2 gigabytes per file. The FAT32 format raised the limit to 4 gigabytes per file. The NTFS format removed the 4 gigabytes limit.

Most external USB storage devices ship with FAT. This is the lowest common denominator for moving data between platforms and is supported by Windows and UNIX systems. Micro focus files that are smaller than 2 gigabytes may easily be moved across all three disk formats. Files greater than 2 gigabytes and less than 4 gigabytes may be easily moved between FAT32 and NTFS. Files that are larger than 4 gigabytes will require the NTFS format.

A separate Document describing Micro Focus File Formats is available on the Internet provides additional information about data file conversion technology.

Table of Contents Data File Migration Process

This section provides an overview of the tasks that will need careful planning in order to ensure a ongoing data migration process (i.e. transfer and conversion) of the data files used by the application.

Table of Contents Archived or Historical Data

This could be a larger task than converting the primary or active data. Some companies are required to maintain years of historical data stored on various formats of tape devices. The two decisions that need to be made are as follows.

Item Description
1 What media will be used to archive historical data?
2 Do we do a conversion from EBCDIC to ASCII of all the archived data or do we do the conversion at a time when it may be necessary to recall the data (there may be some legal or regulatory considerations)?
  Critical Decisions for Archived and/or Historical Data

Table of Contents AS/400 as a Client

If a communication or data sharing requirement with the AS/400 exist then it may be necessary to modify some parts of the application. The AS/400 is an EBCDIC-encoded architecture as is the Mainframe System. Therefore, data that is shared between the Mainframe and the AS/400 is compatible and does not require conversion. However, when moving to an ASCII-encoded environment such as Windows, Linux or UNIX it will be necessary to convert the data that is shared between the two system.

Table of Contents Automated & Repeatable Test Cycle

During an application migration process there are usually numerous testing cycles with different sets of test data and there will be times when the testing produces different results that require analysis, debugging and possible re-testing. This will require multiple conversions of various sets of test and/or production data.

Table of Contents Anticipate Compare during Convert

As part of the final testing process it is common practice to do parallel runs on the mainframe and the new Windows or UNIX platform. After the parallel runs are complete we will be faced with the problem of how to compare the files created or updated on the mainframe with the files created or updated in the new Windows platform. Since the mainframe set of files are EBCDIC-encoded and the Windows (i.e. Micro Focus) files are ASCII-encoded we are faced with another conversion process prior to the file compare process.

Table of Contents Validate & Audit after Completion

The data conversion process should be a repeatable process with an audit trail. The process should be executable as an automated, unattended process. Requiring operator input during the conversion process introduces an exposure point for error. When doing data file content conversion an existing definition (such as a COBOL copy file) of the data structures (or record layouts) should be used. This will avoid introducing errors in a process that requires the data to be defined into a new or proprietary format.

Table of Contents Summary

This document discusses the "possibilities and considerations" aspects of an application migration and will focus on providing an overview of the data management process during a mainframe application migration effort. This document may be used as a tutorial for new programmers or as a quick reference for experienced programmers.

Before we can take these mainframe applications into a production environment the people that will be doing the day-to-day operations and the people that will be providing production support (at the system level or the application level) will need time to develop what we refer to as a minimum experience base. To develop this will require additional training and heavy involvement in the testing process and the support required during the testing process.

Involvement in the support (both problem determination and problem resolution) is key in order to:

Item Description
1 Develop familiarity with the new system.
2 Learn new techniques for analyzing application problems.
3 Developing expertise in using supplied file management tools for both online and batch environment.
4 Learn new techniques for restarting jobs when an application failure occurs.
5 Develop familiarity with the application and how it executes on the new systems platform.
  Skills Development for the New Environment

In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.

The documentation and software were developed and tested on systems that are configured for a SimoTime environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

SimoTime Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SimoTime Services or Technologies please send an e-mail to: helpdesk@simotime.com or call 415 883-6565. We appreciate hearing from you.

Table of Contents Software Agreement and Disclaimer

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.

SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Table of Contents Additional Reference Materials

This section is intended to provide references to additional information for individuals that are migrating applications across systems or share data between systems. Many of the following references will require a connection to the Internet.

Table of Contents Numeric Field Formats

The following items provide external links to reference material for various encoding formats used for numeric data strings.

The following list provides external links to reference material and examples for the various types of numeric formats used with the COBOL programming language and/or the Mainframe System.

Numeric Type  Description
Zoned-Decimal This document describes the zoned-decimal format. This is coded in COBOL as USAGE IS DISPLAY and is the default format if the USAGE clause is missing.
Note: This is the slowest performer and uses the most storage space but is easiest to display on a screen or print to a printer. This encoding scheme may be unsigned (implied positive) or signed. This type of field will require special handling for the sign position when migrating from a mainframe (EBCDIC) to a Micro Focus (ASCII) environment.
Packed-Decimal This document describes the packed-decimal format. This is coded in COBOL as USAGE IS COMPUTATIONAL-3 and is usually coded in its abbreviated form of COMP-3.
Note: The mainframe can perform arithmetic functions with this data format at the hardware (or micro-code) level. This type of encoding scheme was primarily used to save storage space. This encoding scheme may be unsigned (implied positive) or signed. When migrating from a mainframe (EBCDIC) to a Micro Focus (ASCII) environment this type of field should be left in its original format since this will be supported in the new environment.
Binary This document describes the binary format. This is coded in COBOL as USAGE IS COMPUTATIONAL and is usually coded in its abbreviated form of COMP. This may also be coded with the keyword BINARY.
Note: This format will save storage space but was primarily used for performance. Register arithmetic uses this format. This encoding scheme may be unsigned (implied positive) or signed. When migrating from a mainframe (EBCDIC) to a Micro Focus (ASCII) environment this type of field should be left in its original format since this will be supported in the new environment.
Edited Numeric This document describes the edited numeric format. This is coded in COBOL using an edit mask in the picture clause. An example would be PIC ZZZ.99+.
Note: This type of field is used for numbers that are to be displayed or printed and should be all text characters. This field should be converted using standard conversion tables.
Numbers Connection This document provides links to various examples that describe and demonstrate how the various numeric formats are processed.
  Additional Information for the various Types of Numeric Formats used with COBOL

Table of Contents Data File Conversion

The following list provides external links to reference material and examples for data file conversion.

Document Name  Program - Description
dfconv01.htm An overview of the Data File Conversion Possibilities and Considerations
utconv01.htm A utility program to process user-defined specifications and generate COBOL source code for programs that will do File Format and Record Content Conversion
asc2ebc1.htm An ASCII/EBCDIC Table with hex, decimal and binary values.
  Additional Information for Data File Conversion

Table of Contents Data File Comparison

The following list provides external links to reference material and examples for the Data File Comparison process.

Document Name  Program - Description
dfcomp01.htm An overview of the Data File Comparison Possibilities and Considerations
utcomp01.htm A utility program to process user-defined specifications and generate COBOL source code for programs that will compare two data files.
simozaps.htm The Data File Comparison section of the SimoTime Reference Manual is available on the Internet and provides additional information about data file comparison technology.
datcom01.htm The Data File Comparison Self-Study Course provides additional information about using a generated COBOL program to compare two Micro Focus files in a Linux, UNIX or Windows environment. Since the generated COBOL program is OS/390 compliant it may be compiled and executed on an IBM Mainframe with ZOS or VSE.
  Additional Information for Data File Comparison

Table of Contents Data File Transfer

The following items provide external links to reference material and examples for data file transfer.

The following white paper for Transferring Data Files provides additional information about transferring files between a Mainframe and a Wintel platform.

The following is provided as a Reference for the File Transfer Protocol commands. This document describes a typical process for an interactive and automated, batch session.

Table of Contents Non-print or Non-display Characters

Explore  how to remove non-print and non-display characters  in a data string prior to printing or displaying.

Table of Contents Vendors, Consultants & Service Providers

Explore The Micro Focus Web Site for more information about products and services available from Micro Focus. This link requires an Internet Connection.

Table of Contents Glossary of Terms

Link to Internet   Link to Server   Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.

Table of Contents Comments or Feedback

This document was created and is copyrighted and maintained by SimoTime Technologies.

If you have any questions, suggestions, comments or feedback please call or send an e-mail to: helpdesk@simotime.com

We appreciate hearing from you.

Table of Contents Company Overview

SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.

Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.

Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions or need assistance with converting non-relational data structures simply give us a call at 415 883-6565 or check the web site at http://www.simotime.com


Return-to-Top
Application and Data Migration, File Management & Data Integrity
Copyright © 1987-2017
SimoTime Technologies and Services
All Rights Reserved
When technology complements business
http://www.simotime.com