Data Validation Techniques
Review Data Content and Structure
  Table of Contents  v-16.01.01 - datrev01.htm 
  Introduction
  Execute
  Human Observation
  Hexadecimal Dump Format
  Batch Job Run Times
  Batch Job, a JCL Member
  Batch Job, Sample JES Log
  Programmatic File Compare
  A Simple 80/80 Compare
  Customer File, Partial Record Compare
  Summary Totals & Record Counts
  Summary
  Software Agreement and Disclaimer
  Downloads and Links
  Current Server or Internet Access
  Internet Access Required
  Glossary of Terms
  Comments or Feedback
  Company Overview
The SimoTime Home Page 

Table of Contents Previous Section Next Section Introduction

Quite often the Data Validation requirements go beyond a simple file compare. Additional processes (or checkpoints) will need to be put in place to meet the concerns of management and auditors.

When business data is stored in non-relational data structures the physical structures are usually a combination of VSAM Data Sets and traditional sequential files containing records of fixed or variable length. The records are typically concatenated strings of data referred to as fields. A field may contain text strings, binary values (hexadecimal characters or numeric values) or special numeric formats (such as a Packed-Decimal or COMP-3).

Depending on the system environment where the business application is running the data may use an EBCDIC or ASCII encoding schema. An IBM ZOS System and the IBM i-series are typically EBCDIC-encoded. The encoding schema for Windows, UNIX and Linux (LUW) Systems is typically ASCII. Micro Focus Enterprise Server running on an LUW System provides support for both EBCDIC and ASCII encoding.

It is important to note that SimoTime Technologies do not actually do a data file compare or other forms of data validation. The technology generates COBOL programs that will do the actual data validation. The technology (or utility programs) that generate the data validation programs runs on a Windows System and requires Micro Focus Enterprise Developer to be installed. The generated programs may be compiled and executed on a target system that includes Micro Focus on Windows, UNIX and Linux Systems or IBM I-series, ZOS and VSE Systems.

An extra effort or additional verification becomes a requirement for numeric fields that contain currency values. The following sections of this document provide a brief overview of some of the data validation requirements.


We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.

Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved

Table of Contents Previous Section Next Section Execute

This section will focus on the jobs that will do the actual data validation tasks.

The documentation and software were developed and tested on systems that are configured for a SimoTime environment based on the hardware, operating systems, user requirements and security requirements.

Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

Table of Contents Previous Section Next Section Human Observation

Human observation is one of the oldest forms of a data validation technique. The capability of viewing data in a hexadecimal format can be quite useful. The SimoTime Technologies have the ability to generate COBOL programs the will provided hex-dump information at the record level. The generated programs should compile and execute on a Linux, UNIX or Windows System using Micro Focus.

Also, the generated program should compile and execute on a ZOS Mainframe System. The JOB statement and the data set names (DSN's) will need to be changed to comply with security and naming standards on the target system.

Table of Contents Previous Section Next Section Hexadecimal Dump Format

The following link provides an example of a program that will access user-defined records in a VSAM, KSDS and post the Hex-Dump information to a sequential file.

Link to Internet   Link to Server   Explore How to Create Hexadecimal Dump information for the records in a Customer Master file. The Customer Master file is a VSAM, KSDS. This document describes a process for generating a single COBOL program that will read a VSAM/KSDS and write hexadecimal dump information to a sequential file. The user may define the records to be dumped by providing a list of primary keys in a control file. The Hex-Dump information is written to a user-defined sequential file. The Hex-Dump formatting and file I-O are done within the generated COBOL program. The Hex-Dump formatting provides a record structure containing all text characters. Therefore, the contents may be easily viewed using a text editor of choice.

Table of Contents Previous Section Next Section Batch Job Run Times

As part of the Regression Testing process the elapsed execution time for the batch jobs will need to be compared with the execution time of a previous regression testing cycle to insure that performance continues to meet the business expectations.

Link to Internet   Link to Server   Explore how to calculate elapsed times for batch jobs using the JES Log files produced by Micro Focus Enterprise Server.

Table of Contents Previous Section Next Section Batch Job, a JCL Member

The following job was executed on a Windows System with Micro Focus Enterprise Server.

//BCHMRKJ1 JOB SIMOTIME,CLASS=2,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*       BCHMRKJ1.JCL - a JCL Member for Batch Job Processing        *
//*       This JCL Member is provided by SimoTime Technologies        *
//*           (C) Copyright 1987-2019 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//*        TEST MFWAITER EXECUTION WHEN INVOKED VIA JCL
//*        THE FORMAT OF THE WAIT/DELAY PARM IS HH:MM:SS:CC
//*                                          OR HH:MM:SS
//*            HH Hours
//*            MM Minutes
//*            SS Seconds
//*            CC Hundreths of a Second
//* ------------------------------------------------------------------
//STEP0001 EXEC PGM=MFWAITER,PARM='00:00:05'
//* ------------------------------------------------------------------
//STEP0002 EXEC PGM=MFWAITER,PARM='00:00:10'
//* ------------------------------------------------------------------
//STEP0003 EXEC PGM=MFWAITER,PARM='00:00:02'
//*

Table of Contents Previous Section Next Section Batch Job, Sample JES Log

The following is the start time, stop time and elapsed time for the preceding batch job. The job times and the job-step times are provided.

The following is the HTML Formatted JES Log information that is produced when the BCHMRKJ1.JCL job is executed on a Windows System with Micro Focus Enterprise Server.

Date Time JCL Message Job Number Job or Step Condition Job Name Step Name Return Code Elapsed Time
20160329 13:59:19 JCLCM0188I JOB18453 JOB STARTED BCHMRKJ1      
20160329 13:59:19 JCLCM0190I JOB18453 STEP STARTED BCHMRKJ1 STEP0001    
20160329 13:59:24 JCLCM0191I JOB18453 STEP ENDED BCHMRKJ1 STEP0001 0000 00:00:05
20160329 13:59:24 JCLCM0190I JOB18453 STEP STARTED BCHMRKJ1 STEP0002    
20160329 13:59:34 JCLCM0191I JOB18453 STEP ENDED BCHMRKJ1 STEP0002 0000 00:00:10
20160329 13:59:34 JCLCM0190I JOB18453 STEP STARTED BCHMRKJ1 STEP0003    
20160329 13:59:36 JCLCM0191I JOB18453 STEP ENDED BCHMRKJ1 STEP0003 0000 00:00:02
20160329 13:59:36 JCLCM0182I JOB18453 JOB ENDED BCHMRKJ1   0000 00:00:17

The preceding shows the Start, Stop and Elasped time for the entire job and the Start, Stop and Elapsed time for each job step within the job.

Table of Contents Previous Section Next Section Programmatic File Compare

Reviewing the results obtained during a regression test that follows a system, application or programming change is one of the factors that drive a requirement for a programmatic comparison of data files. The scope of this effort is dependent of the type (or format) of file being compared and the complexity of the record structure within the file. Comparing the files is only half of the effort. What to do once an error (or non-equal) condition occurs can be a significant part of the effort.

Table of Contents Previous Section Next Section A Simple 80/80 Compare

The following link is a very simple example of doing a record by record compare of two sequential files.

Link to Internet   Link to Server   Explore a Simple Data File Compare Program that does a record-by-record compare of two sequential files with 80 byte records. The COBOL source code for the program was generated using SimoTime technologies.

Table of Contents Previous Section Next Section Customer File, Partial Record Compare

The following link is an example of doing a record by record compare of two VSAM Keyed-Sequence-Data-Sets (or KSDS). This example does a partial record compare with the positions to be compare defined in a separate, user-defined control file.

Link to Internet   Link to Server   Explore The Generated COBOL Source Code that does a record-by-record compare of two VSAM Key-Sequenced-Data-Sets (KSDS's) that are used as the Customer Master Files.

Table of Contents Previous Section Next Section Summary Totals & Record Counts

It may be a requirement to accumulate batch totals for numeric fields and this may be especially true for currency fields. The SimoTime technology allows a user to leverage various approaches for this requirement.

1. SimoTime provides the technology to do the accumulation of totals for numeric fields based on a record layout and the field definitions provided in a COBOL copy file.
2. SimoTime technology generates COBOL source code that is compiled and then used to do the accumulation of summary totals and record counts.
3. Optional functions for the programs that do the accumulation of summary totals.
3.1. Check for all space characters in a COMP-3 (or packed-decimal) field.
3.2. Check for non-numeric characters in a COMP-3 (or packed-decimal) field.
3.3. Set a flag to correct the previous two conditions.

Link to Internet   Link to Server   Explore how to accumulate Summary Totals and Record Counts by reading a sequential file or VSAM Data Set. This suite of programs and documentation will describe and demonstrate an approach for reading a file, calculating record counts and producing summary totals for a set of pre-defined numeric fields. This technology has the capability of managing non-numeric values in numeric fields.

Table of Contents Previous Section Next Section Summary

This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.

In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.

The documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please send an e-mail to: helpdesk@simotime.com or call 415 883-6565. We appreciate hearing from you.

Table of Contents Previous Section Next Section Software Agreement and Disclaimer

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.

SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Table of Contents Previous Section Next Section Downloads and Links

This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.

Note: A SimoTime License is required for the items to be made available on a local system or server.

Table of Contents Previous Section Next Section Current Server or Internet Access

The following links may be to the current server or to the Internet.

Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the Link to Internet icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the Link to Server icon.

Link to Internet   Link to Server   Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.

Link to Internet   Link to Server   Explore the COBOL Connection for more examples of COBOL programming techniques and sample code.

Link to Internet   Link to Server   Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.

Link to Internet   Link to Server   Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and/or QSAM files.

Table of Contents Previous Section Next Section Internet Access Required

The following links will require an internet connection.

A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection

Explore The Micro Focus Web Site for more information about products and services available from Micro Focus. This link requires an Internet Connection.

Table of Contents Previous Section Next Section Glossary of Terms

Link to Internet   Link to Server   Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.

Table of Contents Previous Section Next Section Comments or Feedback

This document was created and is copyrighted and maintained by SimoTime Technologies.

If you have any questions, suggestions, comments or feedback please call at 1 415 827-7045 or send an e-mail to: helpdesk@simotime.com

We appreciate hearing from you.

Table of Contents Previous Section Next Section Company Overview

SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.

Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.

Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com


Return-to-Top
Data Validation Techniques for review of content and structure
Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved
When technology complements business
http://www.simotime.com