Applications & Data Migration, Management & Data Integrity |
![]() |
The SimoTime Home Page |
Ever since the second computer was introduced into the world the tasks involved with a System Upgrade or replacement, an Application Migration and Data File Management (data file sharing, transfer, conversion and validation or compare) have been technically challenging.
The term "Application Migration" may be defined as the moving or deploying of all the components of an application from an "existing system" to a "new system" and obtaining the same business results. The term "Application" is used to refer to both program members and data structures that include Relational Data Bases, VSAM Data Sets and traditional file structures. The term "system" is used to refer to existing or new hardware, operating system software and sub-systems such as online transaction monitors or job schedulers. The system provides the foundation for the application to execute and deliver the critical business requirements.
If changes become necessary then identifying the "Application Change" tasks and making the changes after a successful migration would help differentiate "risk incurred through application change" vs. "risk incurred through application migration or system dependencies". Also, this process will identify and help to manage the "scope creep" that may occur during an "Application Migration" effort.
Maintaining Data Integrity during an application migration and data file conversion process should be the top priority. To succeed requires careful planning and analysis with constant monitoring of the data transfer, data sharing, data conversion and data validation processes.
This document discusses the "possibilities and considerations" aspects of an application migration and will focus on providing an overview of the data management process during a mainframe application migration effort.
We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.
Copyright © 1987-2023
SimoTime Technologies and Services
All Rights Reserved
The following is a list of the steps for an Application and Data Migration.
1. | Planning and Analysis |
2. | Design, Build and Configure the Receiving System |
2.1. | Hardware |
2.2. | System Software |
2.3. | Sub-Systems and Supporting Software |
3. | Migrate the Application |
3.1. | Transfer and Compile the Code |
3.2. | Transfer and Convert the Data |
3.3. | Unit Test the Application |
4. | Application and System Testing |
4.1. | Proper Program Execution |
4.2. | Data Integrity |
4.3. | Support Structure (Helpdesk) |
5. | Results Validation |
5.1. | Programmatic Data File Compare |
5.2. | Numeric Integrity |
5.2.1. | Accumulate Summary Totals |
5.2.2. | S0C7 or RTS163, Illegal character in numeric field |
5.3. | Human Observation |
6. | Deployment (Monitor and Support) |
When moving from the "application migration and unit testing" phase to the "application and system testing" phase the support requirements will change from a planned, proactive-oriented process to an on-demand, reactive-oriented process. During the migration-oriented phase we had hours or days to plan and resolve issues. As an alternative we could possibly bypass the issue (develop a temporary work around) until a fix was provided.
During the Testing phase individuals doing the testing will be reporting problems as they occur and will be expecting a quick response so they may continue with their testing effort. When a problem is identified in the testing phase it must be resolved quickly with a permanent, long-term fix (a work around or temporary fix with a follow-on permanent fix may invalidate the testing). If an issue is encountered that stops the testing from moving forward it must be resolved within a matter of minutes or hours. If the issue is not resolved in this timeframe it could result in a rescheduling of the testing phase (this could possible require a partial or complete synchronization of the data and other system resources).
Therefore, it is important to have the training for the support personnel completed and involved in the support process prior to starting the "application and system testing". This will provide the primary support required and offer an opportunity to gain hands-on experience with the application running in the new environment.
Many companies have successfully migrated business applications between Mainframe Systems and a Windows, UNIX or Linux Systems. Based on these successes companies are now motivated and actually pursuing the migration of additional applications and data between a Mainframe System and a Windows, UNIX or Linux System.
These additional applications continue to increase in size and complexity along with the transfer, share, convert and compare of data that is formatted in an ever-increasing variety of file structures, formats, encoding schemas, data bases and access methodologies.
The Mainframe System continues to evolve and customers are migrating applications that are an intermingling of reliable, twenty-year old programs and file structures combined with programs and data structures that incorporate the latest mainframe technologies.
Mainframe applications that are created using the COBOL programming language are referred to as "COBOL-oriented applications" in this document. Batch jobs require the use of JCL (Job Control Language) and on-line programs usually interface with a transaction system (for example CICS, Customer Information Control System) and screen handlers such as BMS (Basic Mapping Support).
Therefore, a mainframe application may be predominately COBOL but will have other source members such as JCL or BMS (hence the term COBOL-oriented-application vs. COBOL-application).
In addition, many years ago (prior to the release of the ANSI/85 standard and the release of IBM's COBOL/2 for the mainframe) the COBOL language did not have the functionality to meet some of the programming requirements needed to meet the business and system demands. The COBOL Compilers did not generate executable members that were finite-in-purpose and fast enough to meet the size and performance requirements. To solve these problems Mainframe Assembler programs and callable assembler routines were written and many are still in use today. As the mainframe has evolved the capability of Mainframe Assembler has evolved. Mainframe Assembler continues to be used to address "Systems-oriented" requirements resulting in the combined use of older and newer Mainframe Assembler technologies.
The key to a successful application migration (and obtaining platform flexibility) is the ability to recreate the system and sub-system foundations that will support the execution of the COBOL programs along with the non-COBOL programs and manage the data using the existing application methodologies for a batch or online environment.
The following is an overview of some of the services and training options available for an application migration effort. The following is a list of the recommended consulting, services and training phases designed to minimize the risk and successfully complete an application and data migration effort.
| ||||||||||||||||
Overview of the Phases of an Application and Data Migration Project |
Training will be addressed in the Analysis and Planning phase based on individual assignments and requirements. Each of the options is discussed in more detail in the following sections of this document.
When moving from the "application migration and unit testing" phase to the "application and system testing" phase the support requirements will change from a planned, proactive-oriented process to an on-demand, reactive-oriented process.
During the migration-oriented phase we had hours or days to plan and resolve issues. As an alternative we could possibly bypass the issue (develop a temporary work around) until a fix was provided. During the Testing phase individuals doing the testing will be reporting problems as they occur and will be expecting a quick response so they may continue with their testing effort. When a problem is identified in the testing phase it must be resolved quickly with a permanent, long-term fix (a work around or temporary fix with a follow-on permanent fix may invalidate the testing). If an issue is encountered that stops the testing from moving forward it must be resolved within a matter of minutes or hours. If the issue is not resolved in this timeframe it could result in a rescheduling of the testing phase (this could possible require a partial or complete synchronization of the data and other system resources).
Migrating an application from a mainframe requires lifting the source code for the application off of its current foundation of hardware (mainframe), system software (z/OS or VSE), sub-system (CICS) software and Data Base Management (DB/2) or File Management (VSAM or sequential files) software.
The source code for the application is then shifted to a new foundation of hardware (intel), system software (Windows), sub-system (Micro Focus ES/MTO with Net Express for development) and Data Base Management (MS/SQL) and File Management (Micro Focus for Indexed and Sequential files) software. The source code for the application is then compiled and we are almost ready for our first unit test.
At this point it is important to note that Micro Focus provides the following.
| ||||||||
Micro Focus Capabilities or Alternatives |
Refer to the Micro Focus documentation for more detail.
Third-Party software may be application oriented, sub-system oriented or utilitarian. Many software vendors now offer a Windows or Distributed version of the mainframe software packages. For example, a third-party package that manages and catalogs reports into a database and provides easy access by a business user would be sub-system oriented. A third-party package that provides a file copy function and is called from a JCL submission or command file execution would be utilitarian.
For an "Application and Data Migration" project security requires a two-phased approach. The Micro Focus Enterprise Server, Mainframe Transaction Option (ES/MTO) provides sub-system level security in the Windows environment. To obtain system level security it will be necessary to use a combination of ES/MTO subsystem security and Active Directory Security provided by Microsoft. This is a separate discussion that is beyond the scope of this document.
A number of software vendors including some of the following vendors now offer distributed (or Enterprise) products for Windows, UNIX and Linux (distributed) platforms. If the job scheduling requirements are relatively simple the Task Scheduler provided with Windows may satisfy the scheduling requirement.
ASG-Zena is a robust, enterprise-wide workload management solution for distributed operations environments that support "event-based" scheduling as well as traditional date-and-time-based scheduling methodologies.
BMC Control-M simplifies IT job scheduling across the board. While its features are advanced - automated conversion wizard, customizable self service, application integration - running it couldn't be easier. BMC Control-M is simple to install, simple to get going, and simple for your business to use.
CA Workload Automation enables a user to schedule and manage workload processing across your entire IT environment. Using CA Workload Automation you can update workload processing schedules in real-time to accommodate changing business requirements.
Cisco Tidal Enterprise Scheduler simplifies job scheduling, and automates and tracks how jobs are carried out. Tidal Enterprise Scheduler provides a single point of control and visibility for job scheduling, managing, and performing all scheduling processes.
IBM® Tivoli® Workload Automation enables you to easily manage composite business and IT workloads that have complex dependencies spanning multiple applications, systems including mainframe, distributed and high-performance grid computing environments.
ORSYP technology monitors production through a single management console - and even offers a web interface for remote access. A single enterprise-class automation solution reduces the risk and costs associated with managing and maintaining multiple products and tools while enabling proactive management on critical issues. Automating IT Operations with ORSYP increases productivity, service quality and more efficient usage of existing resources.
UC4 Software delivers a unified automation solution that intelligently orchestrates your business processes, applications, and IT infrastructure. UC4 a single enterprise-wide approach that intelligently integrates the automation of business processes, applications and IT infrastructure onto one platform.
The use of a Job Scheduler will require additional effort when Migrating an application to a distributed platform. This effort usually requires a replacement product (i.e. a vendor's distributed scheduler instead of the vendor's Mainframe scheduler) that will need to be installed and configured for the targeted distributed platform. It is important to note that many job schedulers provide additional capability beyond the simple scheduling of jobs. Jobs may be scheduled as follows.
| ||||||||||||||
Job Scheduling Requirements or Dependencies |
A job scheduler may also do parameter substitution. For example, the use of the %argument syntax may be used as a substitution capability where the substitution value for the %argument is stored in an external table. This capability functions similar to the capability of JES to define and substitute values based on the &argument syntax to identify where the substitution is to take place. The difference is the value for the &argument is specified in the JCL stream and the value for %argument may be specified in an external table or database.
A job scheduler should be able to monitor job progress and take appropriate action based on certain conditions. A job scheduler can also track system usage based on user (or other criteria) that may be necessary for billing.
This section will discuss Report Management Systems and Printing. Third Party Vendors supply sub-system or utilitarian software to extract printed reports from the mainframe output spool and place the reports into a repository for easy access by business users. Mobius Management Systems is one of the vendors that supply this type of software in both a mainframe and a distributed (i.e. Windows) environment.
VPSX® software (provided by Levi, Ray & Shoup, Inc.) is a robust, scalable
Enterprise Output Management System
for UNIX, Linux, Windows, and other distributed environments. Designed to capture data from any platform, VPSX spools that data, then delivers the output to printers, fax servers, file servers, and other devices.
VPSX was designed from the ground up for flexibility and ease of use. When required, VPSX can use external filters to convert data streams from a given source format to the data stream required by the target device or application. Administrators and users can monitor print status, identify and resolve errors, and control the entire printing environment using a flexible Web browser interface.
A data string that is to be printed should not contain binary data (i.e. non-print characters). The binary characters may be the same value as a print control character that could result in the incorrect presentation of the data. For additional information about non-print character processing refer to the Additional Reference Materials section of this document.
This section provides an overview of the tasks that will need careful planning in order to ensure a successful application asset migration (i.e. transfer and compilation) of the source code used to build the application. Most mainframe environments have implemented a Software Configuration Management (or SCM) system.
Depending on the method used to transfer the source code between the Mainframe system and the Windows system it may be necessary to "check-out" a copy of the source members in the SCM to members in a Partitioned Data Set (or PDS). Once the source members are stored in a PDS they may be accessed as EBCDIC-encoded, record sequential files with 80 byte records. The files may be transferred between the mainframe and the Windows system using the File Transfer Protocol (FTP).
Source code members should not contain embedded hexadecimal characters. Refer to Hexadecimal Characters in Source Code for more information about managing embedded hexadecimal characters in source code.
A variety of directory structures or configurations are available in the Windows environment. This section will describe a configuration that works well in the Windows environment and is complimentary to the Micro Focus environment and supports the concurrent existence of a development, test and production environment on a single Windows system or spread across multiple systems. The following is the sub-directory structure used by SimoTime for the sample programs. The primary directory is C:\SimoSAM1.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-1: This is the primary directory for the SimoTime Driver Programs and Test Cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-2: This is the Development, Production or Test directory, for this example it is set to DEVL. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-3: Utility programs using a mainframe dialect, not part of application business functions | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-4: Utility programs using a Micro Focus dialect, not part of application business functions | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-5: Utility programs with specialized or unique requirements | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The Directory Structure used in the in the SimoTime Development Environment |
Additional detail about the structure for the ADM1, DATA, HOLD or SYS1 directories may be accessed by clicking on the "and more" item.
Color Associations: The
The section briefly describes three of the popular methods for transferring source code between a Mainframe System and a Windows, Linux or UNIX System.
A separate Data File Transfer Document is available on the Internet and provides additional information about Data File Transfer process and the alternatives that are available.
When transferring source code from a Mainframe system to a Windows system running a Micro Focus sub-system this should be the preferred transfer method. Since this is a Micro Focus utility the file format conversion (Mainframe file format to Micro Focus file format) is done as part of the file transfer process. The transfer process may be performed on demand using the Graphical User Interface (GUI) with a point-and-click or drag-and-drop methodology. The transfer process may be scripted and automated using a Windows command file, a job scheduler and the MFA command line interface.
This file transfer methodology is usually available on Mainframe systems (both z/OS and VSE), Windows systems, UNIX Systems and Linux systems. Because it is readily available it is usually the default transfer technology.
FTP works quite well when transferring source members that are stored in a Partitioned Data Set (PDS) as sequential files with fixed-length records of eighty characters of printable text. FTP in ASCII mode will do the file transfer, the record content (i.e. EBCDIC and ASCII) and the file format (record sequential to line sequential) conversions.
Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fiber Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.
For the program development environment Micro Focus uses the Microsoft Visual Studio or Eclipse integrated development environments (IDE's) to interactively edit, compile and debug application programs. Information about these products are available from the suppliers.
A scripted build process may be a requirement in larger environments. A predefined and documented process for doing the build of a production system without operator intervention is a minimum requirement for the audit or validation process.
This document will display information to the console and write information to a log file based on the results of the execution of the scripted compile and/or generate processes. The processes will focus on building a test or production system for the ASCII-encoded environment.
This section provides an overview of the tasks that will need careful planning in order to ensure a successful data migration (i.e. transfer and conversion) of the data files used by the application.
The three tasks that are necessary to prepare are listed below and discussed in the following sub-sections of this document.
| ||||||||
Preparing the Environment and Data Files |
The following shows the sub-directory structure for the DATA directories.
Note: The Catalog used by the Micro Focus Enterprise Server is located in the DATA directory.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-1: This directory is identified by the ES_ALLOC_OVERRIDE Environment Variable. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note-2: This directory contains the Micro Focus generated STR (or Structure) Files and the SimoTime generated HTML documents for Record Structures based on a COBOL copy file. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Directory Structure for non-Relational Data Files |
Refer to the Directory Structure for a Development Environment for more information about a complete directory structure
There are a number of possibilities but this section will focus on three common methods for transferring data file.
| ||||||||
Alternatives for Data File Transfers |
A separate Data File Transfer Document is available on the Internet and provides additional information about Data File Transfer process and the alternatives that are available.
This is the transfer methodology used by default because of its availability on most systems. It can easily transfer ASCII/Text files and has the capability to transfer record sequential files in binary mode.
To transfer VSAM, key-sequenced-data-sets (KSDS) requires the file to be converted to a sequential file (using IDCAMS and REPRO). The sequential file is then transferred to the target system. The received file is then used as the base for an EBCDIC to ASCII conversion with the output file being created as a Micro Focus Keyed-Index File.
Mainframe Access (or MFA) provides the simplest method for transferring files from a z/OS mainframe to a Wintel, Micro Focus environment. The "File Format" conversion is handled by the MFA transfer process. The file transfer may be a simple "Drag-and-Drop" task from the GUI interface or an automated, batch process executed from a Windows command line.
Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fiber Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.
For ongoing data sharing and transferring the use of removable media (tape or Compact Disk) that is available and compliant with both the Mainframe system and the Windows system is declining. The use of tape remains the popular media for backup on the mainframe.
The MRM approach usually requires file format conversions to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.
Refer to Additional Reference, Data File Conversion section of this document for additional information about File Format and Record Content conversion.
During an "Application and Data Migration" effort changing file formats should only be done when required by the receiving system or by the file transfer process. The following sections will provide a quick overview of some of the items that need to be considered when doing file format conversions.
These files are also referred to as "Record Sequential" files and may be downloaded (via FTP) in binary format and then converted from EBCDIC to ASCII encoding at the field within record level.
If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file. The received file is then used as the base to convert from EBCDIC to ASCII.
The format of a sequential file with variable length records that resides on the mainframe is very different from a sequential file with variable length records that resides on a Linux, UNIX or Windows System with Micro Focus Enterprise Server.
On the mainframe the length of the individual records within a file is maintained in a Record Descriptor Word (or RDW). Special syntax is required by the FTP process to get the RDW information that contains the length of the individual records within the file. Once the file is received it can only be processed using byte-stream I/O.
If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file. The received file is then used as the base to convert from EBCDIC to ASCII.
If Micro Focus Mainframe Access (MFA) is used this is a simple "Drag-and-Drop" task to transfer the file followed by an EBCDIC to ASCII conversion.
If FTP is used to transfer the file it must first be converted to a sequential file on the mainframe and then the sequential file is transferred to the Windows or UNIX platform. The received sequential file is then used as the base to convert from EBCDIC to ASCII and create a new Micro Focus Indexed file. It is important to note that if the key field is alpha-numeric it will be necessary to provide for the difference in the EBCDIC and ASCII collating sequence.
During an "Application and Data Migration" effort the typical conversion is to change the content of each record in a file from an EBCDIC-encoded format to an ASCII-encoded format. Because the record structures usually contain fields that are not just text strings (i.e. numeric packed-decimal, signed numeric zone-decimal or binary) it is necessary to do the conversion at the field level.
The Micro Focus Data File Converter (DFCONV) is used for the files with record structures that are defined by a single copy file. It is important to note that DFCONV is a developer's tool that is included with Mainframe Express and Net Express. It is not available on UNIX or with Enterprise or Application Server.
The file conversion technology that is available from SimoTime Enterprises is used for the following.
| ||||||||||
List of Possible Systems where the Conversion will need to Execute |
Refer to Additional Reference, Data File Conversion section of this document for additional information about file format and record content conversion.
Doing a conversion between ASCII and EBCDIC is a simple process. Knowing what to convert is the challenge. Understanding the record structure within a file is the key to doing a successful data file conversion at the record and field level. For applications that are currently running on a Mainfame System the definitions for record structures are usually available in the form of a COBOL copy file.
The simple guideline is to convert the alpha-numeric, text-oriented fields and do not convert the binary fields. Numeric fields will need additional considerations.
The three most common numeric types (or formats) are Zoned-Decimal, Packed-Decimal and Binary. Each of the numeric types require special considerations and cannot be converted using the same techniques used for alpha-numeric, text-oriented fields.
Refer to Additional Reference, Numeric Field Formats section of this document for additional information about the various numeric types..
The file comparison technology that is available from SimoTime Enterprises is used to do data file compares on the Windows platform. This technology does not actually do the file compares but generates a COBOL program that performs the file compares. This COBOL program may be compiled and executed on an IBM Mainframe (z/OS or VSE), a Windows platform with Micro Focus or a Linux or UNIX platform with Micro Focus.
Refer to Additional Reference, Data File Comparison section of this document for additional information about comparing data files on a record by record basis.
The Micro Focus Data File Editor is included with Mainframe Express and Net Express. The Data File Editor allows a user to view the contents of a file at the record level using an EBCDIC or ASCII translation of the data. A hex view is also provided and is very helpful if a record within the file contains packed or binary fields.
It is also possible to map record layouts to a COBOL copy file. This can be very helpful for both viewing and modifying data for testing purposes.
This section describes various file formats and the file handling environments used by Micro Focus. For the development environment the base file handler may be the preferred environment. For the Test and Production environment the Micro Focus File Share running on a separate server may be the preferred environment for Indexed Files.
Microsoft has various formats for disk. The FAT (or File Allocation Table) is the older technology used prior to Windows 2000 and many external storage devices are shipped with the initialization being FAT. The FAT format has a limit of 2 gigabytes per file. The FAT32 format raised the limit to 4 gigabytes per file. The NTFS format removed the 4 gigabytes limit.
Most external USB storage devices ship with FAT. This is the lowest common denominator for moving data between platforms and is supported by Windows and UNIX systems. Micro focus files that are smaller than 2 gigabytes may easily be moved across all three disk formats. Files greater than 2 gigabytes and less than 4 gigabytes may be easily moved between FAT32 and NTFS. Files that are larger than 4 gigabytes will require the NTFS format.
A separate Document describing Micro Focus File Formats is available on the Internet provides additional information about data file conversion technology.
This section provides an overview of the tasks that will need careful planning in order to ensure a ongoing data migration process (i.e. transfer and conversion) of the data files used by the application.
This could be a larger task than converting the primary or active data. Some companies are required to maintain years of historical data stored on various formats of tape devices. The two decisions that need to be made are as follows.
| ||||||
Critical Decisions for Archived and/or Historical Data |
If a communication or data sharing requirement with the AS/400 exist then it may be necessary to modify some parts of the application. The AS/400 is an EBCDIC-encoded architecture as is the Mainframe System. Therefore, data that is shared between the Mainframe and the AS/400 is compatible and does not require conversion. However, when moving to an ASCII-encoded environment such as Windows, Linux or UNIX it will be necessary to convert the data that is shared between the two system.
During an application migration process there are usually numerous testing cycles with different sets of test data and there will be times when the testing produces different results that require analysis, debugging and possible re-testing. This will require multiple conversions of various sets of test and/or production data.
As part of the final testing process it is common practice to do parallel runs on the mainframe and the new Windows or UNIX platform. After the parallel runs are complete we will be faced with the problem of how to compare the files created or updated on the mainframe with the files created or updated in the new Windows platform. Since the mainframe set of files are EBCDIC-encoded and the Windows (i.e. Micro Focus) files are ASCII-encoded we are faced with another conversion process prior to the file compare process.
The data conversion process should be a repeatable process with an audit trail. The process should be executable as an automated, unattended process. Requiring operator input during the conversion process introduces an exposure point for error. When doing data file content conversion an existing definition (such as a COBOL copy file) of the data structures (or record layouts) should be used. This will avoid introducing errors in a process that requires the data to be defined into a new or proprietary format.
This document discusses the "possibilities and considerations" aspects of an application migration and will focus on providing an overview of the data management process during a mainframe application migration effort. This document may be used as a tutorial for new programmers or as a quick reference for experienced programmers.
Before we can take these mainframe applications into a production environment the people that will be doing the day-to-day operations and the people that will be providing production support (at the system level or the application level) will need time to develop what we refer to as a minimum experience base. To develop this will require additional training and heavy involvement in the testing process and the support required during the testing process.
Involvement in the support (both problem determination and problem resolution) is key in order to:
| ||||||||||||
Skills Development for the New Environment |
In the world of programming there are many ways to solve a problem. This documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.
SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please contact us using the information in the Contact, Comment or Feedback section of this document.
Software Agreement and Disclaimer
Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.
SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.
This section is intended to provide references to additional information for individuals that are migrating applications across systems or share data between systems. Many of the following references will require a connection to the Internet.
The following items provide external links to reference material for various encoding formats used for numeric data strings.
The following list provides external links to reference material and examples for the various types of numeric formats used with the COBOL programming language and/or the Mainframe System.
| ||||||||||||
Additional Information for the various Types of Numeric Formats used with COBOL |
The following list provides external links to reference material and examples for data file conversion.
| ||||||||
Additional Information for Data File Conversion |
The following list provides external links to reference material and examples for the Data File Comparison process.
| ||||||||||
Additional Information for Data File Comparison |
The following items provide external links to reference material and examples for data file transfer.
The following white paper for Transferring Data Files provides additional information about transferring files between a Mainframe and a Wintel platform.
The following is provided as a Reference for the File Transfer Protocol commands. This document describes a typical process for an interactive and automated, batch session.
Explore how to remove non-print and non-display characters in a data string prior to printing or displaying.
Explore The Micro Focus Web Site for more information about products (including Micro Focus COBOL) and services available from Micro Focus. This link requires an Internet Connection.
Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.
This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.
1. | Send an e-mail to our helpdesk. |
1.1. | helpdesk@simotime.com. |
2. | Our telephone numbers are as follows. |
2.1. | 1 415 763-9430 office-helpdesk |
2.2. | 1 415 827-7045 mobile |
We appreciate hearing from you.
SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.
Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.
Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.
Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com
Return-to-Top |
Application and Data Migration, File Management & Data Integrity |
Copyright © 1987-2023 SimoTime Technologies and Services All Rights Reserved |
When technology complements business |
http://www.simotime.com |