Jesse,
If one end of the pipe fails for any reason, then both ends would have to
be rerun. The redbook "Batch Modernization on
z/OS" chapter 16
explains the use of Batch pipes. Martin Packer is one of the co-author.
http://www.redbooks.ibm.com/abstracts/sg247779.html
Thanks,
Kolusu
From: Jesse 1 Robinson <***@SCE.COM>
To: IBM-***@LISTSERV.UA.EDU
Date: 11/21/2017 03:13 PM
Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
Sent by: IBM Mainframe Discussion List <IBM-***@LISTSERV.UA.EDU>
I have never worked with pipes in a production batch environment. In a
classic batch job, if a problem occurs somewhere in the middle, it's
common to restart the job at some recoverable point after fixing the
downstream problem. I cannot visualize how to restart a pipes job in such
a case. Without intermediate files to resume with, how does all that
in-memory data get recreated without starting back at the top?
In a short or medium length job, starting over may not be such a big
problem, but for a job that runs hours on a good day, I could envision
some major production schedule glitches with pipes.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
***@sce.com
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On
Behalf Of Sri h Kolusu
Sent: Tuesday, November 21, 2017 1:54 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
Hi,
Here is a sample DFSORT job that reads a pipe and does a sort while
modifying the data using inrec(before sorting) Outrec(after sorting) and
Outfil
//PIPEDEMA JOB (DA26,001,098,J69),'KOLUSU',
// CLASS=A,
// MSGCLASS=H,
// MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//**************************************************************
//* DELETE THE INPUT PIPE DATASET IF IT EXISTED **
//**************************************************************
//STEP0001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//DD1 DD DSN=&SYSUID..PIPE.INPUT,
// UNIT=SYSDA,SPACE=(TRK,0),DISP=(MOD,DELETE)
//*
//**************************************************************
//* CREATE THE INPUT TO SORT AND WRITE IT TO A PIPE **
//* WILL HAVE 60 RECORDS (20 RECORDS FOR EACH NAME) **
//**************************************************************
//PIPERCRE EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD *
PAUL
MARTIN
HOBART
//SORTOUT DD DSN=&SYSUID..PIPE.INPUT,
// SUBSYS=(BP01,'WAITOPEN=180'),
// DCB=(LRECL=80,RECFM=FB,BLKSIZE=27920)
//SYSIN DD *
OPTION COPY
OUTFIL REPEAT=20,OVERLAY=(20:SEQNUM,3,ZD,RESTART=(1,10))
/*
//PIPEDEMB JOB (DA26,001,098,J69),'KOLUSU',
// CLASS=A,
// MSGCLASS=H,
// MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//**************************************************************
//* READ THE PIPE AND USING INREC MODIFY MARTIN RECORDS **
//* SORT THE DATA ON NAME AND SEQUENCE DESCENDING **
//* MODIFY PAUL RECORDS USING OUTREC **
//* MODIFY HOBART RECORDS USING OUTFIL **
//**************************************************************
//PIPEREAD EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTDIAG DD DUMMY
//SORTIN DD DISP=SHR,DSN=&SYSUID..PIPE.INPUT,
// SUBSYS=(BP01,'WAITOPEN=180'),
// DCB=(LRECL=80,RECFM=FB,BLKSIZE=27920)
//SORTOUT DD SYSOUT=*
//SYSIN DD *
INREC IFTHEN=(WHEN=(01,10,CH,EQ,C'MARTIN',AND,
20,03,SS,EQ,C'005,010,015,020'),
OVERLAY=(31:C'EVERY 5TH RECORD TAGGED USING INREC'))
SORT FIELDS=(01,10,CH,A, $ SORT ON NAME ASC
21,03,ZD,D) $ SORT ON SEQ DESC
OUTREC IFTHEN=(WHEN=(01,10,CH,EQ,C'PAUL',AND,
20,03,SS,EQ,C'006,012,018'),
OVERLAY=(31:C'TAGGED EVERY 6TH RECORD USING OUTREC'))
OUTFIL IFTHEN=(WHEN=(01,10,CH,EQ,C'HOBART',AND,
20,03,CH,EQ,C'020'),
OVERLAY=(31:C'RECORD 20 IS TAGGED USING OUTFIL'))
/*
Further if you have any questions please let me know
Thanks,
Kolusu
DFSORT Development
IBM Corporation
Date: 11/21/2017 01:06 PM
Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
Post by Martin PackerA DFSORT stage would/could/should allow all the myriad* DFSORT
control
statements. And be for heavy lifting. Those are the two reasons I
suggested
it.
I was going to do some details research (my DFSORT knowledge is
probably out of date, for which I take responsibility), but time is
short, and I don't want the issue to get stale. I believe you are
referring to INCLUDE/OMIT, INREC/OUTREC, etc.
I hope this answers your point. I don't know if your statement is
true,
or
will ever be implemented. I will say that Pipelines offers so many
more options in record selection, formatting, conversion and field
specification, that what you suggest probably will never need to be
implemented.
- Of the 300+ Pipelines stages, many are selection stages. In this
case,
"many" is much more than myriad, AFAIK, by an order of magniture.
There are simple selectors, but also complex ones that support
conversions, logic, arithmetic, and reformatting.
- Fields can be identified by character, word, or delimited field.
Each
of
these can used as substring to an enclosing specification, to multiple
levels. E.g., take the second slash delimited subfield in the seventh
tab
delimited field to get a numeric month value. And you are not limited
to
blanks as white space or tabs as delimiters.
- The DFSORT program product control statements can only be used in
connection with the sort process. Pipelines stages can be used even
when
you are not sorting. In other words, Pipelines skills, while being
easy
to
learn (unlike my recollection of DFSORT control statement), they have
a vastly broader range of applicable uses.
- The DFSORT program product E15 and E35 internal sort exits can only
be used once per sort. Pipelines stages bring this efficiency to each
and every stage in a Pipeline. To press the point, unlike the need
for
record
and field definitions in, say, a COBOL internal sort exit, the
intermediate
records don't have to be defined, and fields only have to be
referenced
if
they are actually involved in the a particular stage.
- Finally, you can process unselected records without having to
reprocess
the entire input file, as shown below.
Business problem: You have an externally produced name and address
file that has to be sorted by ZIP code to get a USPS discount rate.
There
are
two types of records, business and residential. The fields are free
format
and tab delimited. The business records differ from residential
records
in
that there are additional fields: Company after the name fields and
an
ATTN: field after the ZIP code. This would be trivial in pipes if the
ZIP
field.
A pipes approach is as follows: Separate the records into streams and
prefix each record with its ZIP code. Then sort by the ZIP in the
first field, discard the prefix ZIP. It would look something like
this in commented REXX. ( Visualizing someone else's multi-stream pipe
is easier with a flow chart.)
...
/*--------------------------------------------------------*/
/* f200: f800: */
/* +-+ +-+ +----+ +-+ +------+ +----+ +-+ */
/* |<|-->|p|-->|spec|-->|f|-->|dfsort|-->|chop|-->|>| */
/* +-+ |i| +----+ |a| +------+ +----+ +-+ */
/* |c| |n| */
/* |k| +----+ |i| */
/* | |-->|spec|-->|n| */
/* +-+ +----+ +-+ */
/*--------------------------------------------------------*/
"pipe (listerr name ZIPSort.exec end ? ",
"| < in.file ",
"| f200: pick 1.1 == /R/", /* Residential records */
"| specs field 8 1 1-* nextfield", /* ZIP pfx */
"| f800: fanin ", /* Concat residential and business */
"| dfsort option nolist sort fields=(5,10,cha,a)",
, /* NOT compound stage causes CHOP to keep the orig. rcd.*/
"| not chop after 05", /* Trim prefix key and tab.*/
"| > out.file */
"? f200: ", /* Business record come here. */
"| specs field 9 1 1-* nextfield", /* ZIP pfx */
"| f800: ", /* Back to first stream. */
...
I hope this helps. Even if there are features new (to me) of DFSORT
would
make some of my statements incorrect, I'm quite sure that the
remaining ones would still drive the point home easily.
On Sat, Nov 18, 2017 at 3:01 AM, Martin Packer
Post by Martin PackerA DFSORT stage would/could/should allow all the myriad* DFSORT
control statements. And be for heavy lifting. Those are the two
reasons I
suggested
Post by Martin Packerit.
And note a DFSORT stage could be tested outside of a pipeline.
Also note Pipelines could orchestrate multiple DFSORT invocations in
a chain.
Cheers, Martin
* If you don’t think the word “myriad” is appropriate RTFM. :-) The
point
Post by Martin Packeris there’s lots of processing capability there.
Sent from my iPad
Post by Hobart SpitzLet me clarify a couple of things.
The PIPEs SORT stage only uses virtual storage. The DFSORT stage
is
used
Post by Martin PackerPost by Hobart Spitzwhen your data is larger than the virtual storage available.
DFSORT
uses
Post by Martin PackerPost by Hobart Spitzdisk also, and runs a bit slower.
TSO Pipelines and BatchPipesWorks are two versions/names of
CMS/TSO Pipelines that runs under z/OS. I have used both at a
number of
sites.
Post by Martin PackerPost by Hobart SpitzAFAIK, they are still available. They are both chargeable.
Someone
will
Post by Martin PackerPost by Hobart Spitzcorrect me if I'm wrong, but NetView pipes is a different animal
and
I
Post by Martin PackerPost by Hobart Spitzdoubt it has any significant bearing on this discussion.
On Fri, Nov 17, 2017 at 2:29 PM, Jesse 1 Robinson <
Post by Jesse 1 RobinsonAh Netview Access. Some years ago we had a SHARE (JES2) Singalong
ditty
Bunch.
Post by Martin PackerAll
Post by Hobart SpitzPost by Jesse 1 Robinsonabout a family (of software products) bound together without a
common
Post by Martin PackerPost by Hobart SpitzPost by Jesse 1 Robinsonparent but designated siblings through a marriage of convenience.
But alas, I also doubt that Netview Pipes could be implemented
outside
already
Post by Martin Packerhave
Post by Hobart SpitzPost by Jesse 1 RobinsonNetview for other business purposes, acquiring it just for Pipes
might
Post by Martin Packerbe a
Post by Hobart SpitzPost by Jesse 1 Robinsontough sell.
.
.
J.O.Skip Robinson
Southern California Edison Company Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
-----Original Message-----
From: IBM Mainframe Discussion List [
On
Post by Hobart SpitzPost by Jesse 1 RobinsonBehalf Of Farley, Peter x23353
Sent: Friday, November 17, 2017 11:07 AM
Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
Dumb question of the day: Assuming you have NetView installed,
can
the
a
Post by Martin Packerbatch
Post by Hobart SpitzPost by Jesse 1 Robinsonstep that executes IRXJCL or IKJEFTxx to invokes a Rexx program?
Based on the discussion so far I am guessing probably not, but
thought
Post by Martin Packerit
Post by Hobart SpitzPost by Jesse 1 Robinsonwas worth throwing out the question.
I am totally ignorant of NetView functionality other than using
NetView
Post by Martin PackerPost by Hobart SpitzPost by Jesse 1 RobinsonAccess for certain 3270 terminal logons, so please be gentle.
Peter
-----Original Message-----
From: IBM Mainframe Discussion List [
On
Post by Hobart SpitzPost by Jesse 1 RobinsonBehalf Of Jesse 1 Robinson
Sent: Friday, November 17, 2017 1:26 PM
Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
I suspected that this function belonged to Netview, but as you
noted, it
Post by Martin PackerPost by Hobart SpitzPost by Jesse 1 Robinsonwas implemented here only during SA installation. I brought it up
because
Post by Hobart SpitzPost by Jesse 1 Robinsonthere is obviously native z/OS code involved. Porting that to TSO
might
Post by Martin Packerbe
Post by Hobart SpitzPost by Jesse 1 Robinsona tad easier than lifting code across from z/VM.
.
.
J.O.Skip Robinson
Southern California Edison Company Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
-----Original Message-----
From: IBM Mainframe Discussion List [
On
Post by Hobart SpitzPost by Jesse 1 RobinsonBehalf Of Nims,Alva John (Al)
Sent: Friday, November 17, 2017 7:09 AM
Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base.
Thank you for "PIPing" up on this! :)
One thing to note, the PIPE command is part of the Base NetView
product,
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN