Data Validation Techniques Review Data Content and Structure

Introduction

Quite often the Data Validation requirements go beyond a simple file compare. Additional processes (or checkpoints) will need to be put in place to meet the concerns of management and auditors.

When business data is stored in non-relational data structures the physical structures are usually a combination of VSAM Data Sets and traditional sequential files containing records of fixed or variable length. The records are typically concatenated strings of data referred to as fields. A field may contain text strings, binary values (hexadecimal characters or numeric values) or special numeric formats (such as a Packed-Decimal or COMP-3).

Depending on the system environment where the business application is running the data may use an EBCDIC or ASCII encoding schema. An IBM ZOS System and the IBM i-series are typically EBCDIC-encoded. The encoding schema for Windows, UNIX and Linux (LUW) Systems is typically ASCII. Micro Focus Enterprise Server running on an LUW System provides support for both EBCDIC and ASCII encoding.

It is important to note that SimoTime Technologies do not actually do a data file compare or other forms of data validation. The technology generates COBOL programs that will do the actual data validation. The technology (or utility programs) that generate the data validation programs runs on a Windows System and requires Micro Focus Enterprise Developer to be installed. The generated programs may be compiled and executed on a target system that includes Micro Focus on Windows, UNIX and Linux Systems or IBM I-series, ZOS and VSE Systems.

An extra effort or additional verification becomes a requirement for numeric fields that contain currency values. The following sections of this document provide a brief overview of some of the data validation requirements.

We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.

Copyright © 1987-2025
SimoTime Technologies and Services
All Rights Reserved

Execute

This section will focus on the jobs that will do the actual data validation tasks.

The documentation and software were developed and tested on systems that are configured for a SimoTime environment based on the hardware, operating systems, user requirements and security requirements.

Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

Human Observation

Human observation is one of the oldest forms of a data validation technique. The capability of viewing data in a hexadecimal format can be quite useful. The SimoTime Technologies have the ability to generate COBOL programs the will provided hex-dump information at the record level. The generated programs should compile and execute on a Linux, UNIX or Windows System using Micro Focus.

Also, the generated program should compile and execute on a ZOS Mainframe System. The JOB statement and the data set names (DSN's) will need to be changed to comply with security and naming standards on the target system.

Hexadecimal Dump Format

The following link provides an example of a program that will access user-defined records in a VSAM, KSDS and post the Hex-Dump information to a sequential file.

    Explore How to Create Hexadecimal Dump information for the records in a Customer Master file. The Customer Master file is a VSAM, KSDS. This document describes a process for generating a single COBOL program that will read a VSAM/KSDS and write hexadecimal dump information to a sequential file. The user may define the records to be dumped by providing a list of primary keys in a control file. The Hex-Dump information is written to a user-defined sequential file. The Hex-Dump formatting and file I-O are done within the generated COBOL program. The Hex-Dump formatting provides a record structure containing all text characters. Therefore, the contents may be easily viewed using a text editor of choice.

Batch Job Run Times

As part of the Regression Testing process the elapsed execution time for the batch jobs will need to be compared with the execution time of a previous regression testing cycle to insure that performance continues to meet the business expectations.

    Explore how to calculate elapsed times for batch jobs using the JES Log files produced by Micro Focus Enterprise Server.

Batch Job, a JCL Member

The following job was executed on a Windows System with Micro Focus Enterprise Server.

//BCHMRKJ1 JOB SIMOTIME,CLASS=2,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*       BCHMRKJ1.JCL - a JCL Member for Batch Job Processing        *
//*       This JCL Member is provided by SimoTime Technologies        *
//*           (C) Copyright 1987-2019 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//*        TEST MFWAITER EXECUTION WHEN INVOKED VIA JCL
//*        THE FORMAT OF THE WAIT/DELAY PARM IS HH:MM:SS:CC
//*                                          OR HH:MM:SS
//*            HH Hours
//*            MM Minutes
//*            SS Seconds
//*            CC Hundreths of a Second
//* ------------------------------------------------------------------
//STEP0001 EXEC PGM=MFWAITER,PARM='00:00:05'
//* ------------------------------------------------------------------
//STEP0002 EXEC PGM=MFWAITER,PARM='00:00:10'
//* ------------------------------------------------------------------
//STEP0003 EXEC PGM=MFWAITER,PARM='00:00:02'
//*

Batch Job, Sample JES Log

The following is the start time, stop time and elapsed time for the preceding batch job. The job times and the job-step times are provided.

The following is the HTML Formatted JES Log information that is produced when the BCHMRKJ1.JCL job is executed on a Windows System with Micro Focus Enterprise Server.

Date
Time
JCL Message
Job Number
Job or Step
Condition
Job Name
Step Name
Return Code
Elapsed Time


20160329
13:59:19
JCLCM0188I
JOB18453
JOB
STARTED
BCHMRKJ1
 
 
 


20160329
13:59:19
JCLCM0190I
JOB18453
STEP
STARTED
BCHMRKJ1
STEP0001
 
 


20160329
13:59:24
JCLCM0191I
JOB18453
STEP
ENDED
BCHMRKJ1
STEP0001
0000
00:00:05


20160329
13:59:24
JCLCM0190I
JOB18453
STEP
STARTED
BCHMRKJ1
STEP0002
 
 


20160329
13:59:34
JCLCM0191I
JOB18453
STEP
ENDED
BCHMRKJ1
STEP0002
0000
00:00:10


20160329
13:59:34
JCLCM0190I
JOB18453
STEP
STARTED
BCHMRKJ1
STEP0003
 
 


20160329
13:59:36
JCLCM0191I
JOB18453
STEP
ENDED
BCHMRKJ1
STEP0003
0000
00:00:02


20160329
13:59:36
JCLCM0182I
JOB18453
JOB
ENDED
BCHMRKJ1
 
0000
00:00:17

The preceding shows the Start, Stop and Elasped time for the entire job and the Start, Stop and Elapsed time for each job step within the job.

Programmatic File Compare

Reviewing the results obtained during a regression test that follows a system, application or programming change is one of the factors that drive a requirement for a programmatic comparison of data files. The scope of this effort is dependent on the type (or format) of files being compared and the complexity of the record structure within the files. Comparing the files is only half of the effort. What to do once an error (or non-equal) condition occurs can be a significant part of the total effort.

A Simple 80/80 Compare

The following link is a very simple example of doing a record by record compare of two sequential files.

    Explore a Simple Data File Compare Program that does a record-by-record compare of two sequential files with 80 byte records. The COBOL source code for the program was generated using SimoTime technologies.

Customer File, Partial Record Compare

The following link is an example of doing a record by record compare of two VSAM Keyed-Sequence-Data-Sets (or KSDS). This example does a partial record compare with the positions to be compare defined in a separate, user-defined control file.

    Explore The Generated COBOL Source Code that does a record-by-record compare of two VSAM Key-Sequenced-Data-Sets (KSDS's) that are used as the Customer Master Files.

Summary Totals & Record Counts

It may be a requirement to accumulate batch totals for numeric fields and this may be especially true for currency fields. The SimoTime technology allows a user to leverage various approaches for this requirement.

    Explore how to accumulate Summary Totals and Record Counts by reading a sequential file or VSAM Data Set. This suite of programs and documentation will describe and demonstrate an approach for reading a file, calculating record counts and producing summary totals for a set of pre-defined numeric fields. This technology has the capability of managing non-numeric values in numeric fields.

Summary

This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.

In the world of programming there are many ways to solve a problem. This documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please contact us using the information in the  Contact or Feedback  section of this document.

Software Agreement and Disclaimer

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.

SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Downloads and Links

This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.

Note: A SimoTime License is required for the items to be made available on a local system or server.

Current Server or Internet Access

The following links may be to the current server or to the Internet.

Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.

    Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.

    Explore the COBOL Connection for more examples of COBOL programming techniques and sample code.

    Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.

    Explore The File Status Return Codes that are used to interpret the results of accessing VSAM data sets and/or QSAM files.

Internet Access Required

The following links will require an internet connection.

A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection

Explore The Micro Focus Web Site for more information about products (including Micro Focus COBOL) and services available from Micro Focus. This link requires an Internet Connection.

Glossary of Terms

    Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.

Contact or Feedback

This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.

We appreciate hearing from you.

Company Overview

SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.

Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.

Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com