Data File Transfer
Data Management Series
|The SimoTime Home Page|
This document is an introduction or overview of the data file transfer aspects of an application migration between a mainframe system and a Windows system running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express.
The data file transfer process should be a repeatable process with an audit or validation trail. The process should be executable as an automated, unattended process. Requiring operator input during the data file transfer process introduces an exposure point for error.
A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.
We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.
Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved
There are two categories of application (i.e. source code and associated control/configuration information) and data file transfer requirements based upon the application migration requirements.
|1.||Migrate or move the application to a Micro Focus environment on a Linux, UNIX or Windows platform and then retire the application from executing on the Mainframe|
|1.1.||Move the application assets and data to the new platform. This may be considered a one-time migration. However, the approach should be to create a repeatable process for many of the tasks. For the larger migration efforts the source code and data may need to be refreshed multiple times over the duration of the migration and testing cycles.|
|1.2.||This is usually a one-way conversion|
|2.||Replicate or copy the application to a Micro Focus environment on a Linux, UNIX or Windows platform and then coexist with the mainframe by sharing data and processes.|
|2.1.||This may be an on-going transfer/exchange/conversion of data|
|2.2.||This may require bi-direction translation (i.e. download/upload)|
|3.||Select one of the preceding categories.|
If you are not familiar with the Micro Focus file formats this link will provide additional detail of the files supported in a Micro Focus Server environment.
The following questions will need to be answered in order to determine the scope of effort for transferring the non-relational data files.
|Defining the Data Transfer Methodology|
When transferring files from a Mainframe system to a Windows system running a Micro Focus sub-system Mainframe Access (MFA is provided by Micro Focus) is the preferred transfer method. Since this is a Micro Focus utility the file format conversion (Mainframe file format to Micro Focus file format) is done as part of the file transfer process.
The transfer process may be performed on demand using the Graphical User Interface (GUI) with a point-and-click or drag-and-drop methodology. The transfer process may be scripted and automated using a Windows command file, a job scheduler and the MFA command line interface.
Additional consideration is required for very large files (greater than 2 gigabytes). The solution does require the MFA server component to be installed on a Mainframe system running z/OS and is not available for Mainframe systems running VSE.
The following is an example of the command line statement required to download a VSAM, KSDS from the Mainframe system to a Windows system running Micro Focus Mainframe Express. The file will be converted to a Micro Focus Indexed file as part of the transfer process. This is a single statement but is shown on two lines.
C:\> MFDAS IMPORT BINARY c:\SimoDEMO\DataMfa1\Simotime.data.itemmast.dat FROM kids simotime.data.itemmast
Note-1: the file will be downloaded in its original EBCDIC-encoded format. The numeric fields will be maintained in their mainframe numeric formats (this includes packed, binary and floating point).
Note-2: If a conversion from EBCDIC to ASCII is required this will need to be done as a separate task.
If the file is larger than 2 gigabytes then extra configuration settings or environment variables will be required.
C:\> set MFA_INDEX_FORMAT=IDXFORMAT(8) C:\> set extfh=c:\SimoDEMO\ProdLib1\ExtfhBig.CFG
The ExtfhBig.CFG file contains the following records.
[XFH-DEFAULT] filemaxsize=8 idxformat=8 filepointersize=8
The preceding should allow for files that are greater than 2 gigabytes in size to be downloaded.
This file transfer methodology is usually available on Mainframe Systems (both z/OS and VSE), Windows Systems, UNIX Systems and Linux Systems. Because it is readily available it is usually the default transfer technology.
When transferring files between a Mainframe System and a Windows System the FTP technology works well with record sequential files of fixed-length records of printable text (i.e. no binary data such as numeric fields defined as PACKED or BINARY). For sequential files with variable length records additional FTP statements are required in order to preserve the Record Descriptor Word (the RDW is a binary value that precedes each record and defines the record length).
FTP works quite well when transferring source members that are stored in a Partitioned Data Set (PDS) as sequential files with fixed-length records of eighty characters of printable text. FTP in ASCII mode will do the file transfer, the record content (i.e. EBCDIC and ASCII) and the file format (record sequential to line sequential) conversions.
FTP, by default, will run in ASCII mode and when transferring files between a Mainframe system and a Windows, UNIX or Linux System it will attempt to convert the record content between EBCDIC encoding and ASCII encoding. Running in ASCII mode will also convert the file format from record sequential to line sequential. This will create a problem if the record content contains binary data.
FTP has the capability of transferring files in a BINARY mode without changing the content of the records. This mode is especially useful if the records in a file contain binary data (i.e. numeric fields that use the packed, binary or signed-zoned-decimal format).
For sequential files that contain EBCDIC-encoded text strings and binary text strings and reside on the mainframe the BINARY mode will transfer the file to a Windows or UNIX platform as a record sequential file. The EBCDIC-encoded text strings and the binary text strings are not converted. In a Micro Focus environment it is possible to run in EBCDIC mode.
However, if information is to be imported into a non-Micro Focus environment such as an Excel spreadsheet the text strings will need to be converted from EBCDIC to ASCII and the binary strings (typically packed or binary numbers) will need to be expanded (i.e. converted) to printable text. If the numeric value is a signed number the sign will need to be placed in a separate byte at the beginning or end of the number. If the numeric value has decimal positions then an explicit decimal point may need to be inserted.
For non-sequential files this approach usually requires a file format conversion to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.
Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fiber Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.
For ongoing data sharing and transferring the use of removable media (tape or Compact Disk) that is available and compliant with both the Mainframe system and the Windows system is declining. The use of tape remains the popular media for backup on the mainframe.
The MRM approach usually requires file format conversions to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.
This section provides a list of questions that will aid in determining the scope of effort for creating the process to do data file transfers. The following questions should be answered at the beginning of the process to migrate an application and its associated data.
The following is used to determine the number of files and the file types and characteristics.
|1.||How many Key-Sequenced-Data-Sets (KSDS or Indexed Files) do you have to be transferred?||______|
|1.a||How many of these files have one or more alternate indexes?||______|
|1.b||Determine maximum file size.||______|
|1.b.1||How many records in the largest file?||______|
|1.b.2||What is the record size?||______|
|2.||How many Sequential files do you have to be converted?||______|
|2.a||How many files are fixed record length?||______|
|2.b||How many files have variable length records?||______|
|3.||Will you be using Line Sequential (i.e. ASCII/Text) files in the Windows environment?||Y or N|
This section provides additional detail about the process for the transfer of data files between a mainframe system and a Windows system running a Micro Focus sub-system. The file transfer technology and methodology that is used will determine the amount of file conversion (or preparation) that is required prior, during or after the file transfer.
Both MFA and FTP do not transfer the alternate indices. Alternate indices must be created after the file has been downloaded. Building the alternate index may be done using IDCAMS and JCL (Mainframe Express or the Batch Facility with ES/MTO) or with the Micro Focus BLDINDEX utility.
Note: If an EBCDIC to ASCII conversion is required the alternate index should not be created until after the file has been converted. Since we are changing the value (i.e. encoding format) of the alternate key (or index) then we should only create the alternate key after the data that makes up the alternate key has been converted.
Both MFA and FTP will transfer files with variable length records. MFA will do the file format conversion from the Mainframe format (i.e. each record has a Record Descriptor Word or RDW) to a Micro Focus format (i.e. the file has a 128 byte header and each record is preceded by a Record Descriptor Word or RDW). It is important to note that the RDW used on the Mainframe is a different structure than the RDW format used with Micro Focus.
The following is an overview of the mainframe format types for files with variable length records.
|V||Variable, if not specified the default value is 'V".
The four bytes preceding the logical record is the Record Descriptor Word. The content is as follows.
The four bytes preceding the logical record (Descriptor Word) are as follows.
|VBS||Variable Blocked Spanned
The four bytes preceding the logical record is the Segment Descriptor Word. The content is as follows.
The following is a sample FTP script to retrieve (or GET) a sequential file with variable-length records from a Mainframe system to a Windows system. This FTP script is for using at the client or Windows system.
userid password CD .. PWD BINARY QUOTE SITE RDW GET mainframe.dataset.name drive:\directory\filename.ext QUIT
The file must be transferred in BINARY. This will maintain the EBCDIC-encoding and the mainframe numeric formats for packed-decimal, binary and zoned-decimal data.
One of the following FTP statements must be used prior to a GET or PUT statement in order to download the Record-Descriptor-Word (RDW) and the possible Block-Descriptor-Word (BDW). The BDW and RDW are in binary format so it is critical to download in binary even if the records in the file are all text strings. Prior to a GET use one of the following statements.
QUOTE SITE RDW or LITERAL SITE RDW
If the FTP process is to be executed on the mainframe the GET statement will need to be replaced with a PUT and the QUOTE or LITERAL statement will need to be replaced with the following statement.
Once the file is transferred it is a mirror of the mainframe format with EBCDIC-encoding. Each record is preceded by the Record Descriptor Word (RDW) and possible Block Descriptor Word (BDW) and must be converted to a Micro Focus formatted sequential file with variable length records. This process is described in the following sections of this document.
The following will provide an overview of the approaches and considerations for managing Sequential File with Variable Length records.
|1.||Variable Length Files, Mainframe Systems with ZOS or VSE.|
|1.1.||Each record is preceded by a four byte Record Descriptor Word (RDW).|
|1.2.||The format types are specified in the DD statement with the RECFM=V or RECFM=VB.|
Variable Length Files, LUW Systems with Micro Focus Enterprise Server.
Note: LUW is an acronym for Linux, UNIX or Windows.
|2.1.||The First record in the file is a unique Micro Focus Header.|
|2.2.||Each subsequent record is preceded by a record descriptor field that is the length of the record.|
|3.||File Transfer Alternatives for Files with Variable Length Records.|
|3.1.||FTP (File Transfer Protocol) is a common transfer methodology that is used to copy files between systems of the same or different architectures.|
|3.1.1.||Typically limited to Sequential Files|
|3.1.2.||The FTP default transfer mode is ASCII.|
|184.108.40.206.||When transferring an EBCDIC-encoded Record Sequential File from a Mainframe System to an LUW System FTP will do a File Format Conversion between Record Sequential and Line Sequential (ASCII/Text) and a record content conversion between EBCDIC and ASCII.|
|220.127.116.11.||The record content must be a text string with valid ASCII characters.|
|18.104.22.168.||If the record content contains numeric values that use a packed or binary format they will be improperly converted.|
|3.1.3.||The FTP transfer mode may be changed to BINARY.|
A physical image of the file will be transferred.
Note: as mentioned in a previous section of this document additional statements are required to get the RDW information.
|22.214.171.124.||The record structure and EBCDIC-encoding are not changed.|
|126.96.36.199.||The numeric values for packed and binary formats are maintained.|
|3.1.4.||Files with Variable Length Records will require special handling.|
|188.8.131.52.||Refer to the Micro Focus VRECGEN functionality for converting the file formats.|
|184.108.40.206.||Refer to the "Downloads and Links to Similar Pages" section of this document for additional SimoTime Technologies for converting the file formats.|
|3.2.||MFA (Mainframe Access) is the preferred transfer methodology that is used to copy files between a Mainframe System with ZOS and an LUW System with Micro Focus Enterprise Developer.|
|3.2.1.||MFA will transfer Sequential Files and VSAM Data Sets.|
|220.127.116.11.||MFA will do a file format conversion between the ZOS format and the Micro Focus File format.|
|18.104.22.168.||MFA will not modify the record content. Typically, the records are EBCDIC-encoded and may contain packed or binary fields with numeric values.|
|3.2.2.||MFA will transfer an Entire Partitioned Data Sets (PDS's) or individual members within a PDS.|
|22.214.171.124.||MFA will do a file format conversion between the ZOS Record Sequential format (usually EBCDIC-encoded, 80-byte, fixed-length records) and the Windows Line Sequential format (usually ASCII-encoded, variable-length records with a maximum record length of 80-bytes).|
|126.96.36.199.||MFA will modify the record content. Typically, the records are EBCDIC-encoded and will be converted to an ASCII-encoded format. The records should not contain embedded hexadecimal or binary values.|
When using FTP to transfer a VSAM, Key-Sequenced-Data-Set (KSDS) from a Mainframe System to a Windows System it will be necessary to read the KSDS and write to a sequential file and then transfer the sequential. The following member (CUSK2RJ1.jcl) is a two (2) step job. The first step will use IEFBR14 with a DD statement to delete a file that was created by a previous execution of this job. The second step will use IDCAMS and the REPRO function to read the Customer Master File and create a sequential file with the same record structure. and content.
//CUSK2RJ1 JOB SIMOTIME,ACCOUNT,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1 //* ******************************************************************* //* CUSK2RJ1.JCL - a JCL Member for Batch Job Processing * //* This JCL Member is provided by SimoTime Technologies * //* (C) Copyright 1987-2019 All Rights Reserved * //* Web Site URL: http://www.simotime.com * //* e-mail: firstname.lastname@example.org * //* ******************************************************************* //* //* TEXT - COPY (OR REPRO) A KSDS TO A SEQUENTIAL FILE //* AUTHOR - SIMOTIME TECHNOLOGIES //* DATE - JANUARY 01, 1989 //* //* ******************************************************************* //* Step 1 of 2, Delete any previously created file... //* //QSAMDELT EXEC PGM=IEFBR14 //SEQ51201 DD DSN=SIMOTIME.DATA.CUSRE512,DISP=(MOD,DELETE,DELETE), // SPACE=(TRK,(10,1),RLSE), // DCB=(RECFM=FB,LRECL=512,DSORG=PS) //* //* ******************************************************************* //* Step 2 of 2, Create and populate a new QSAM file... //* //REPROSEQ EXEC PGM=IDCAMS //SYSPRINT DD SYSOUT=A //KSD51201 DD DSN=SIMOTIME.DATA.CUSTMAST,DISP=(SHR) //SEQ51201 DD DSN=SIMOTIME.DATA.CUSRE512, // SPACE=(TRK,(10,1),RLSE), // DISP=(NEW,CATLG,DELETE), // DCB=(RECFM=FB,LRECL=512,DSORG=PS) //SYSIN DD * REPRO - INFILE(KSD51201) - OUTFILE(SEQ51201) /*
Using Micro Focus Mainframe Access it is a simple process to transfer a single member in a PDS or to transfer the entire PDS and do an EBCDIC to ASCII conversion if required. Other file transfer methods may be able to easily transfer a single PDS Member but will require all the members in a PDS to be extracted and placed in a single sequential file that will be transferred to the target system. Also, this will require the individual PDS members in the sequential file to be rebuilt as individual members on the target system.
The "How to Extract Multiple Source Members" documents in the Downloads and Links to Similar Pages section of this document will provide additional documentation about expanding the sequential file that contains all of the PDS members.
Note: A user will need to be aware of what EBCDIC/ASCII conversion default and configuration options are available during the file transfer processing.
This is part of the basic requirements to ensure the files being transferred will have an equal number of records sent from the Mainframe system and received at the Windows system.
Management and/or the auditors/consultants should be involved in the data migration (transfer, conversion and validation) process as early as possible. This will help ensure the process will meet the requirements and maintain the necessary level of data integrity at each step in the process.
When transferring data files a process for insuring the number of records sent from the mainframe system is received at the Windows system. This is usually accomplished by accumulating record totals on both systems and comparing these control totals. A secondary validation is to accumulate hash totals on both systems for critical currency fields and compare the totals. The totals for currency fields may already be available from standard reports created by the application.
The data file transfer process should be a repeatable process with an audit trail. The process should be executable as an automated, unattended process. Operator intervention during the data file transfer process should be considered as an exposure for introducing errors into the process.
Anyone considering a data migration should seek additional assistance from a consulting or programming services organization that has experience in this area. This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.
In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.
The documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.
SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please contact us using the information in the Comments or Feedback section of this document.
Software Agreement and Disclaimer
Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.
SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.
This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.
Note: A SimoTime License is required for the items to be made available on a local system or server.
The following links may be to the current server or to the Internet.
Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.
Explore How to FTP (using BINARY mode) a File with Variable Length Records from a Mainframe System to a Windows System and then convert the Mainframe format to a Micro Focus format. The mainline program will call the SIMOVREC and SIMOBSIO modules that are available from SimoTime in a separate package.
Explore a Series of White Papers for non-relational data files. This includes information about data file management in a diverse or mixed systems environment.
Explore a quick overview of the data file management tasks for data file transfer, conversion and comparison. Ever since the second computer was introduced into the world the file management tasks of data file transfer, share, convert and compare (or data file validation) have been technically challenging.
Explore the alternatives for transferring data files between systems. This link provides access to a repository of information that includes the transferring and/or sharing of data between Mainframe (ZOS or VSE), Linux, UNIX and Windows Systems.
Explore the Principles of Data File Conversion. This link includes guidelines for defining requirements and determining the scope of effort for a data conversion effort.
Explore the Principles of Data File Comparison. This link includes guidelines for defining requirements and determining the scope of effort for a data comparison effort.
Explore the File Transfer Protocol (FTP) commands using an interactive or scripted batch interface. This document describes a typical process for an interactive and automated, batch FTP session running on a Windows System and connecting to another Windows System, a Linux or UNIX System or an IBM Mainframe System.
Explore How to Extract Multiple Source Members contained in a single sequential file into a folder (or directory) on a Windows System. Each member in the sequential file is delimited by a -PUNCH record.
Explore How to Extract Multiple Source Members contained in a single sequential file into a folder (or directory) on a Windows System. Each member in the sequential file is delimited by a START and END record.
Explore How to Extract Multiple Source Members contained in a single sequential file into a folder (or directory) on a Windows System. The sequential file was created on a Mainframe System using IEBPTPCH.
Explore the non-Relational Data Connection for more examples of accessing methodologies and coding techniques for Data Files and VSAM Data Sets.
Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.
Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and/or QSAM files.
Explore The Binary or COMP format for numeric data strings. This numeric structure is supported by COBOL and may be explicitly defined with the "USAGE IS COMP" or "USAGE IS BINARY" clause.
Explore The Edited for Display format for numeric data strings. This numeric structure is supported by COBOL and may be used with an edit-mask to prepare the presentation for readability by human beings.
Explore The Packed-Decimal or COMP-3 format for numeric data strings. This numeric structure is supported by COBOL and may be explicitly defined with the "USAGE IS COMP-3" clause.
Explore The Zoned-Decimal format for numeric data strings. This numeric structure is the default numeric for COBOL and may be explicitly defined with the "USAGE IS DISPLAY" clause.
Explore commonly used formats and processing techniques for managing various numeric formats available on the mainframe.
Explore the Numbers Connection for additional information about the structure and processing of numeric data items (or numeric fields).
The following links will require an internet connect.
A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection
Explore The Micro Focus Web Site for more information about products and services available from Micro Focus. This link requires an Internet Connection.
Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.
This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.
|1.||Send an e-mail to our helpdesk.|
|2.||Our telephone numbers are as follows.|
|2.1.||1 415 763-9430 office-helpdesk|
|2.2.||1 415 827-7045 mobile|
We appreciate hearing from you.
SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.
Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.
Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.
Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com
|Data File Transfer, Data Management Series|
|Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved
|When technology complements business|